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ABSTRACT 



The purpose of this study was to develop performance 
measures that will assist federal agencies to assess the quality and 
usefulness of their World Wide Web sites and to improve public information 
access service of those Web sites. The study was sponsored jointly by three 
federal agencies: the Defense Technical Information Center, the Energy 
Information Administration, and the Government Printing Office (GPO) . The 
investigators conducted a literature review, reported in Chapter 1. They 
undertook an analysis of laws and policies affecting Web site performance 
measures, reported in Chapter 2. They met with agency representatives 
individually and collectively and analyzed documents provided by the 
agencies, as Chapter 3 explains. Chapter 4 describes the performance measures 
that emerged from the study. Chapter 5 sets forth key issues and recommends 
next steps. Appendices include: criteria for the evaluation of federal Web 
sites; Web site performance measurement and evaluation for GPO Access; and a 
proposal to study agency compliance with federal Web site policies. (Contains 
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PREFACE 



The U.S. Energy Information Administration, the Government Printing Office, and the 
Defense Technical Information Center funded the study that resulted in the information 
reported here. The study was conducted between March, 2000 and September, 2000. 
Given the federal government’s emphasis on developing E-services, E-commerce, and E- 
resources via the networked environment, these three agencies took a leadership position 
to support a study that would assist them (and hopefully other agencies) to better assess 
the services provided via their websites. 

The authors appreciate the assistance participants from these agencies provided in 
developing assessment techniques, reviewing their agency procedures and activities, and 
working directly with the investigators. This effort, however, should be considered as a 
first step in developing assessment techniques and specific performance measures to 
assess web based services. 

The investigators wish to acknowledge the assistance of others who contributed to 
completion of this study. Bruce Fraser, Bruce Smith, and Goldie Burton, all at Florida 
State University, Information Use Management and Policy Institute, contributed to early 
drafts of portions of this report. 

The investigators believe that the findings presented in this study will be an important 
means to move forward in developing future assessment techniques and performance 
measures. The rapidly changing networked and web environment means that assessment 
techniques and performance measures must also continue to evolve. Beyond the efforts 
reported here, much work remains to be done in this area. 

Charles R. McClure 
J. Timothy Sprehe 
Kristen Eschenfelder 

October 1, 2000 
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Performance Measures for Federal Agency Websites 
EXECUTIVE SUMMARY 



1.0 INTRODUCTION 

As federal agencies move to the Web environment as a primary means to disseminate 
information to the public, they require performance measures concerning the extent to 
which their websites are successfully presenting and conveying the government 
information the public needs to access and use. Performance measures for rating the 
success of federal agency websites in meeting their goals are consistent with the 
provisions of the Paperwork Reduction Act Amendments of 1995, the Clinger-Cohen 
Act, the Government Performance and Results Act (GPRA), and OMB Circular No. A- 
130. Foremost among these is GPRA, which mandates the adoption of a strategic and 
annual planning process tied to budget and authorization cycles and based on established 
and measurable performance indicators for every program. 

The purpose of this study was to develop performance measures that will assist agencies 
to assess the quality and usefulness of their websites and to improve public information 
access services of those websites. The study was sponsored jointly by three federal 
agencies: the Defense Technical Information Center, the Energy Information 
Administration, and the Government Printing Office. The investigators conducted a 
literature review, reported in Chapter 1 . They undertook an analysis of laws and policies 
affecting website performance measures, reported in Chapter 2. They met with agency 
representatives individually and collectively and analyzed documents provided by the 
agencies, as Chapter 3 explains. Chapter 4 describes the performance measures that 
emerged from the study. Chapter 5 sets forth key issues and recommends next steps. 

The study employed a multi-method approach to data collection and analysis. Given the 
relatively short time line for the study and the limited resources available the following 
approaches were employed: 

• Literature Review. The study reviewed selected literature and reports from both 
inside and outside the government. 

• Group Interviews with Participating Agency Representatives. The 

investigators (1) met individually with representatives of each participating 
agency, (2) met with the participants as a group, and (3) conducted a number of 
individual follow-up discussions with representatives of the three agencies about 
activities and procedures for assessment in their agency. 

• Best Practices. Each of the three participating agencies provided the 
investigators with (1) descriptions of their assessment techniques, and (2) 
documentation and reports related to these assessment techniques; these materials 
were reviewed and analyzed by the investigators. 

• Policy Analysis. The investigators reviewed existing federal policies, laws, and 
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guidelines to produce an overview of policies affecting the development and 
management of federal websites. 

• Survey. The investigators conducted a survey at the FedWeb 2000 conference 
held in March, 2000 at Bethesda, MD, that provided a snapshot of current 
practices and key issues of concern to federal officials regarding the assessment of 
federal websites. 

• Participant Review. Officials from the three participating agencies had a 
number of opportunities to review and discuss preliminary findings and reports 
developed by the investigators. 

Website evaluation can be defined as the use of research or investigative procedures to 
systematically investigate the effectiveness of a web based information system on an 
ongoing basis. Evaluation is used in conjunction with a number of activities: 

• Planning, Goal Setting and Determining the Degree to Which an Information 
Systems Organization is Meeting Set Goals 

• Decision-making and Resource Allocation 

• Determine the Effectiveness of a Previous Decision 

• Determine the Degree to which the System/Service Adds Value to the 
Organization 

• Provide Trend Data to Assess Change Over Time 

• Contribute to Continuous Improvement Efforts and Benchmarking 

• Identify Problems and Possible Solutions 

• Empower Organizational Actors to Seek and Enact Solutions 

• Develop Accountability 

• Organizational Learning 

Performance measures are the tools that enable evaluation. Performance measures 
typically provide measurements of the following: 

• Inputs and Efficiency. The use of resources in providing or accessing web 
services. 

• Effectiveness. How well the networked information service met the objectives of 
the provider, user or customer. 

• Outputs. Indicators of the products or services resulting from the use of those 
resources. 

• Extensiveness. How much of a service the network provides. 

• Service Quality. How well a service or activity is done. 

• Impact. How a service made a difference in some other activity or situation. 

• Usefulness. The degree to which the services are useful or appropriate for 
individual. 

• Adoption. The extent to which institutions or users integrate and adopt web 
resources in organizational or individual activities. 

One can see federal agency web system evaluation as a special subset of web information 
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systems (IS) evaluation that in turn is a subset of general IS evaluation. IS evaluation has 
become an increasingly important topic within the competitive U.S. business 
environment. The last five years has seen a surge of interest in website evaluation. 

Federal website evaluation has been ongoing since the inception of federal websites. One 
early landmark was the World Wide Web Federal Consortium publication of suggested 
guidelines for federal website development. 

In summary, website evaluation is an important part of the ongoing management and 
enhancement of federal agency websites. Evaluation, whether as part of a formal 
planning process or as a stand-alone activity, can provide managers with key information 
that can aid decision making. Although web sites are a relatively new medium, IS 
evaluation has a long history, providing tools and lessons learned that federal web 
managers can apply to their web evaluation efforts. Current academic research on federal 
agency web evaluation also provides strategies and tools that agencies cam adapt for their 
efforts. 

2.0 FEDERAL POLICIES AFFECTING WEBSITE DEVELOPMENT AND 
MANAGEMENT 

Federal information policies establish the legal and procedural framework in which 
government information and services are made available to the public. The study provides 
an introductory review of selected U.S. Federal information policy instruments that affect the 
development and management of Federal websites. This listing is not comprehensive, but rather 
offers a general sense of the range of existing Federal policies that should be considered in 
the development, management, and evaluation of websites. Issues that Federal agencies need 
to consider in developing websites include information security, information privacy, 
information access, electronic records management, and intellectual property. 

Table 2.1 is a summary presentation of the study’s elaboration of federal policies affecting 
agency websites. For each topic the table shows the relevant statute, executive order, or 
other key document, plus implementing policy guidance, if any. The right hand column 
summarizes the implications for websites. The text of Chapter 2 summarizes the policy 
instruments and includes pertinent quotations from the instruments themselves. 

3.0 REVIEW OF AGENCY PRACTICES 

The report provides an overview of evaluation and performance measures practices at the 
Defense Technical Information Center (DTIC), the Energy Information Administration 
(EIA), and the Government Printing Office (GPO) current as of July 2000. On March 15 
and 16, 2000, the investigators interviewed agency personnel from each of the 
participating federal agencies in site visits to the agencies. Each of these agencies 
supplied documents in answer to the investigators' queries. 

From each agency, the study team requested the following: 
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Table 2.1 Summary of Federal Policies Pertaining to Agency Websites 





Topic 


Statute, Presidential 
Directive, or Other 
Document 


Implementing 

Guidance 


Website Implications 


I. General 


A. Performance and 


Government Performance 


OMB Circular A-ll, 


Performance plans, goals, and 


Government 


Results 


and Results Act 


Part II 


measures for agency programs 


Policy 


B. Customer Service 


E.O. 12862, Setting 
Customer Service Standards 


- 


Identify customers, their needs, 
and set standards and benchmarks 




C. Accessible 

Information 

Technology 


Rehabilitation Act, section 
508 




Information technology accessible 
to persons with disabilities 




D. Electronic 
Government 


Pres. Memo on Electronic 
Government 




Standardized access to and ease of 
finding government information, 
plus privacy and security 


II. Federal 

Information 

Policy 


A. National 
Information 
Infrastructure 


Nil Agenda for Action 




Make govt, information more 
easily and equitably accessible 


B. Privacy and the 
Nil 


Principles for Providing and 
Using Personal Information 


- 


Guidelines to personal 
information users and providers 




C. Copyright 


Digital Millennium Copyright 
Act 


- 


Protecting copyright in electronic 
media 




D. Rights of Access 
to Information 


Freedom of Information Act 


- 


State FOLA procedures on 
websites 






Electronic Freedom of 
Information Act 


- 


Establish electronic reading room 
on websites 






Privacy Act 


OMB Circular A- 130, 
Appendix I 


Handling of personal information 








Pres. Memo on 
Privacy and Personal 
Information in 
Federal Records 


Review privacy policies and 
practices; update notices of 
systems of records 








M-99-18 on Privacy 
Policies on Federal 
Websites 


Display privacy policies on 
websites 








M-00-13 on Privacy 
Policies and Data 
Collection on Federal 
Websites 


Discouragement of and 
restrictions on use of “cookies” 
on websites; comply with COPPA 






Public Printing and 
Documents: Depository 
Library Program 




Publications provided to 
depository library program. 




E. Paperwork 
Reduction Act 


Paperwork Reduction Act 


OMB Circular A- 130 


Framework for agency 
information management plan, 
including information 
dissemination 




F. Clinger-Cohen 
Act 


Information Technology 
Management Reform Act 


E.O. 13011, Federal 

Information 

Technology 


Websites to be interoperable and 
standardized across government 




G. Security 


Computer Security Act 


Proposed Revision of 
OMB Circular A- 130, 
April 13, 2000 


Mission based performance 
measures for information systems 








OMB Circular A-130, 
Appendix HI 


Security controls for federal 
information systems 








PDD 63, Protecting 
America’ Critical 
Infrastructures 


Performance measures for website 
security 




H. Electronic 
Collection and 
Digital Signatures 


Government Paperwork 
Elimination Act 


OMB Notice: 
Implementation of the 
Government 
Paperwork 
Elimination Act 


Increase and encourage electronic 
data collection and implement 
digital signatures 




I. Federal Records 


Federal Records Act 


General Records 
Schedule 20, 
Electronic Records, 
and various NARA 
Bulletins 


Provide for management of 
records created on websites 
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I. Set of documents by which the agency evaluates its website material. 

II. What types of statistics and performance measures is the agency currently 
using for its website? 

III. What overall evaluation methodologies are currently in place and how are 
they conduct ed/operated? 

IV. Can we determine the types of reports generated by the agency evaluation 
efforts and the success or quality of these reports? 

V. To what degree are information policy instruments/guidelines (as per the 
compilation in I) being considered in the documents and reports? 

To the above questions, the investigators added: 

VI. Preliminary review of website 

VII. Summary comments 

The objectives of the review and assessment were to: 

• Identify the basic evaluation methodologies, statistics, and performance 
measures currently in use at each of the agencies 

• Provide a composite view of the three agencies that would allow each to 
compare its practices and techniques with others 

• Consider which activities might constitute “best practices” for federal website 
performance measures 

• Establish a first step toward developing statistics and measures for federal 
website performance measures. 

DTIC. The DTIC website acts as a portal for the public to other sites designed for fuller 
public use. Further, it provides fuller service to users who are technologically 
sophisticated and familiar with the agency’s structure, context, and background. In the 
past, DTIC has relied on technical self-evaluation sources such as log file analysis, 
bandwidth usage and tracked web site response time. But recent efforts have also 
included user based evaluation techniques such as the Secure STINET customer 
satisfaction survey. DTIC is also involved in a larger CENDI Agency working group 
effort to incorporate greater use of web evaluation tools. 

EIA. The task faced by EIA in putting its information on the web is an organizational 
conundrum. The EIA web committee has the unenviable assignment of posting highly 
diverse technical information from many sources on one website suitable for use by the 
layperson. To this end, and to their credit, the EIA web committee has recognized the 
need for continual evaluation and redesign of the site in reaction to a constantly changing 
environment. They have conducted studies to aid in the creation of the next version of 
their home page to be unveiled in the near future. The new design takes a channel 
approach, strives for less jargon, and is designed with the layperson in mind. To prepare 
for this new design, EIA chose to perform cognitive usability testing. The results from 
this usability testing suggest that EIA’s current web site design meets key usability 
requirements as most users could find the required information. 
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GPO. GPO established the GPO Access website to fulfill the mandate required by PL 
103-40. Under the provisions of this law it has subsequently evolved to provide public 
access to many key government documents. As of November 1999, the website provided 
access to more than 104,000 titles on Government Printing Office servers and over 
62,000 additional titles through links to other federal agency websites. GPO considers 
Permanent Public Access to be an important performance measure for federal agency 
websites. GPO Access maintains historical archives of information previously available 
on its servers, ensuring permanent public access to its collection of electronic government 
information products, and facilitating the same historical research that is possible through 
printed media. 

As a content oriented web site, GPO Access primarily measures the number of 
documents retrieved by users as a selection of the units of content being distributed by 
the site. In addition, staff constantly evaluate other aspects of the website through focus 
groups, online user surveys, user comments, log analysis, trade shows and conferences. 
Further, GPO conducts user-training sessions. GPO has ensured that the information on 
its website is listed in commercial search engines. GPO also has conducted bandwidth 
usage studies to determine peak user times and peak usage pages. GPO has a well 
developed management approach for its website and has committed significant effort and 
thought to evaluating it. 

The evaluation and measurement efforts at DTIC, ELA, and GPO demonstrate the wide 
range of strategies for measurement. The efforts also suggest that different strategies can 
be useful in different agency settings. For example, some agencies may be better able to 
employ regular and automatic log analysis techniques than to implement full-scale 
usability studies. Others may find that the establishment of formal usability labs to “test” 
the effectiveness of selected pages and content may be more effective. Ultimately, the 
“best” strategies may be those that best accommodate agency mission and available 
resources. 

The review of agency practices described here may also be useful for other federal 
agencies to compare themselves against. It is not that one agency is better or worse than 
another; rather, that these agencies have taken different approaches to developing 
evaluation and measurement techniques. A range of issues and situational factors affect 
the success with which any agency can engage in an ongoing program of evaluation and 
measurement. 

Each of these three agencies is in the process of developing and evolving their assessment 
techniques. Within this process each has struggled with a comprehensive planning 
approach to formally establish responsibilities, tasking, schedules, statistics, and 
measures to be employed to assess the quality of agency websites and their usefulness to 
users of that website. 
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4.0 PROPOSED PERFORMANCE MEASURES 



The report describes and elaborates on the findings from the assessment of the materials 
collected from the three sponsoring agencies concerning the types of statistics and 
performance measures the agencies were using. Appendix A, Criteria for the Evaluation 
of Federal Websites, and Appendix B, Website Performance Measurement and 
Evaluation for GPO Access, supplement the information about performance measures. 

As the investigators sifted through the data collected from the agencies and from their 
own analyses, they realized that the term “performance measure” can be understood in 
several ways. Many things lumped under the general heading of performance measures 
might be considered conditions prerequisite to performance measures. In other cases, the 
materials pertained to management or operational considerations rather than 
measurement of the website’s performance. Performance measures necessarily imply 
performance goals and the measures yield indicators of progress toward achieving the 
goals. 

The study divided performance measures into three types: 

1 . Legal and policy conditions affecting agency websites 

2. Management and infrastructure factors 

3. Performance measures in the stricter sense of the term 

These three types of performance measures are summarized below in Tables 4.1, 4.2, and 
4.3. 
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Table 4.1. Checklist of Legal and Policy Conditions 
Pertaining to Federal Agency Websites 



Statute/Policy 


Checklist Question 


1. Privacy 


A. Does the website contain a privacy notice that complies with the OMB guidance 
and model language for federal websites? 

B. Does the website avoid the use of “cookies” or observe OMB-stipulated 
restrictions? 

C. Does the website comply with the Children’s Online Privacy Protection Act, 
particularly with regard to collecting personal information from children? (Same as 
9-B below) 


2. Freedom of 
Information Act 


A. Conventional FOIA: Does the website contain clear procedures for requesting 
agency records under FOIA? 

B. Electronic FOIA: Does the website contain an Electronic FOIA Reading Room? 


3. Copyright 


A. Does the website management include provisions to ensure that copyrighted 
materials are not posted without permission from copyright holders? 

B. Reuse Restrictions: Can the site content be freely reused without restriction? 


4. Accessibility 


Does the website make provision for accessibility for persons with disabilities in 
accordance with §508 of Rehabilitation Act? 


5. Security 


Does the website management include adequate provisions for protecting the 
security of the website and other agency information systems? 


6. Paperwork 
Reduction Act 


A. Do information collections undertaken via the website have appropriate OMB 
clearances? 

B. Does the website comply with provisions for the Government Information 
Locator Service? 


7. Depository 
Library Program 


Have publications posted to the website been made available to the Federal 
Depository Library Program? 


8. Government 
Paperwork 
Elimination Act 


A. Does the website permit and encourage electronic information collection/ 

B. Does the website permit use of digital signatures? 


9. Federal Records 
Act 


Does the website management include adequate provision for identifying website 
records and transferring records to agency record keeping systems? 


10. Access for 
Children 


A. Does the website comply with the President’s April 1997 guidance on 
expanding Internet access for children, parents, and teachers? 

B. Does the website comply with the Children’s Online Privacy Protection Act, 
particularly with regard to collecting personal information from children? (Same as 
1-C above) 
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Table 4.2. Management and Infrastructure Factors 
Pertaining to Federal Agency Websites 



Type of Factor 


Factor 


1. Infrastructure 


A. Does the website have sufficient technology infrastructure (bandwidth, 
server controller array, etc.) to ensure adequate response time? 


B. Does the website have the appropriate software to achieve its purposes? 


C. Does the website have the appropriate level of availability (e.g., 24 hours, 7 
days a week)? 


D. Does the website have adequate steps for emergency preparedness and 
disaster recovery? 


E. Does the website have adequate operating resources (funding and personnel) 
to achieve its purposes? 


F. Are server errors and 404 errors (pages not found errors) at an acceptably 
minimum level? 


2. Management 


A. Does the website comport with industry best practices, such as the World 
Wide Web Federal Consortium’s Home Page Guidelines, as regards content 
design, navigation, organization, style, and markup. 


B. Access control: Does the agency exert management control over who may 
post to the website? 


C. Quality control: Does the agency exert management control over the quality, 
design, and style of website postings? 


D. Awareness and Visibility: Has the agency ensured the site is easily reachable 
by registering with search engines and portals and periodically searching for its 
sites on these engines? 


E. User Support: Has the agency provided an adequate user support staff for 
technical and content queries? 


F. Security: Is the agency providing adequate security to prevent intrusion into 
the website? 


G. Evaluation: Does the agency regularly evaluate the website through log file 
analysis, focus groups, online user surveys, open public forums, and other 
appropriate methodologies? 


H. Records Management: Has the agency made adequate provision for records 
management on the website and integrated the website into the agency records 
management program? 
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Table 4.3. Performance Measures for Federal Agency Websites 



Performance Goal 


Basic Measures 7 * • 


Other Agency Specific Measures jj J 


Extensiveness: 
Amount or extent 
to which services 
are used 


-Information on content 
unit retrievals; e.g., no. of 
document downloads 
-No. of user sessions per 
time period (not no. of hits) 


-No of User contact sessions 

-Activity levels by time periods 

-Ratio of unique to repeat (2 or more) user sessions 

per time period. 


Efficiency : Use of 
resources in 
providing services 


-Cost of providing website 
session per user 
-Percent of operational 
time when website is not 
available 


-No. of FTE hours or days devoted to website 
creation/ maintenance by size of site in pages 
-Cost per user help session 

-Relation to diminishing costs of other publications 
media (e.g., printing) as indicator that website may 
be replacing other media. 


Effectiveness: How 
well the website 
meets the general 
governmental 
objectives and 
specific agency 
objectives 


-Completeness of coverage 
of agency publications, 
press releases, etc. 

-Degree to which website 
is increasing the timeliness 
of access to agency pubs 


-Permanent public access to agency publications 
-Degree to which GILS is integrated into website 
design/operations 

-Degree to which website shows agency reaching 
new constituent audiences 


Service Quality: 
How well the 
website functions 


-User success rate in 
finding specific 
information in a given time 
period 

-Average time between 
user contact request and 
agency response 
-No. of customer 
complaints/suggestions and 
whether agency action 
results. 


-Whether agency has Help Desk dedicated to its 
website 

-24/7 availability measures 
-Minimal 404 errors 

-Courtesy, helpfulness of user support staff 
-Increase in no. of repeat users per time period 


Usefulness: How 
well the website 
meets the needs of 
users 


-Customer comments plus 
surveys and focus groups 
-Degree to which website 
information increases user 
productivity 

-Degree to which website 
information is incorporated 
into other tasks inside and 
outside the agency 


-Cognitive and Usability Evaluation 
-Measured user satisfaction with: 

♦Clarity of homepage; organization of site 
♦Timeliness of website information 
♦Links to other useful information 
-No. of referrals from other websites and sources of 
referrals 



In conclusion, the investigators offer the following recommendations: 

1. Federal agencies should address the questions in Table 4.1 regarding agency 
websites. These are basic questions about whether the practices of the agencies’ 
sites comply with federal law and policy. 

2. Agencies should consider the questions in Table 4.2 as a useful but 
nonexhaustive checklist of infrastructure and management questions they should 
pose with respect to their websites. 
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3. Agencies should ask themselves whether their website performance measures 
programs address the six performance goals listed in Table 4.3 and their 
associated basic measures. The investigators do not necessarily recommend that 
websites use the exact measures listed in the table, but they do recommend that 
all agencies employ measures at least similar to these. 

5.0 ISSUES, RECOMMENDATIONS, AND NEXT STEPS 

The issues discussed in this study are not intended to be comprehensive; rather, they suggest 
important concerns that will need to be reviewed, discussed, and resolved at individual 
agencies. The solution to these issues is likely to vary depending on a range of situational 
factors in each agency. 

Key issues agencies must address and resolve concerning websites include the following 

• Top Level Administrative Support for Website Assessment 

• Ongoing Program of Website Evaluation 

• Organizing for Website Assessment 

• Funding Website Assessment 

• Agreement on Approach and Measures 

• Training 

• Recognizing the Policy Context 

• Sharing Knowledge 

• A Changing Web Evaluation Context 

Federal agency website evaluation and development of performance measures is in its 
infancy and much work lies ahead in designing, testing, and implementing evaluation 
methods and measures. The investigators recommend two broad areas for next steps that 
can build upon the work described in this report. 

• Undertake a follow-on study to explore in greater depth some of the issues 
surfaced in this research. Appendix C provides an example of a proposal for such 
a study. 

• Disseminate findings widely and increase awareness. This study is limited in scope 
and yet its findings could prove useful to many other agencies. 

Evaluation and performance measures to describe federal websites and resources in the 
networked environment are important tools. But an ongoing program of evaluation that 
regularly produces these performance measures is a tool that is essential for the long-term 
success of the agency. Perhaps most importantly, an ongoing program of evaluation 
contributes to the process of constant improvement - looking for ways to improve the 
usefulness, impact, and benefits that can result from web-based resources and services. At 
issue is the degree to which these web-based resources and services are cost effective, deliver 
high-quality services, meet the needs of users, comply with existing policy, reduce agency 
costs, and help accomplish agency mission and objectives - to name but a few criteria. 
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CHAPTER 1 



INTRODUCTION 



1.0 BACKGROUND 



Federal agency web sites increasingly act as key information and service resources for 
the agencies themselves, citizens, public interest groups, government employees, 
contractors and other stakeholders. Agency web managers have seen web page use go 
“up up up” as more people access more federal agency web pages more often. Five years 
ago, in the nascent stages of federal web site development, web mangers could evaluate 
their web system based on perceived usage and expansion. As long as usage was 
increasing, and agency personnel were putting more information and services on line, 
“things were good.” 

Today’s environment, however, demands more sophisticated measures for web site 
evaluation. IT resource demands are increasing, federal performance review guidelines 
demand increased efficiency, users grow more sophisticated and demand more 
information and more services. This environment of growing demands and constricting 
resources requires evaluation and performance measures that allow web managers to do 
more than show that usage is going “up up up” and that the overall size of the web site is 
increasing. Today’s federal agencies need evaluation and performance measures to aid in 
more complex tasks such as: 



• Increasing site usability, content and service quality; 

• Creating strategic plans and prioritizing projects; 

• More efficiently and effectively allocating resources; 

• Improving information and service quality and; 

• Enhancing behind the scenes management processes. 



In addition, a variety of federal laws, regulations, and guidelines prescribe how websites 
are to be managed and evaluated (see Chapter 2). Foremost among these is the 
Government Performance and Results Act of 1993 (GPRA), which mandates the 
adoption of a strategic and annual planning process tied to budget and authorization 
cycles and based on established and measurable performance indicators for every 
program. 



Websites are now a major vehicle through which federal agencies deliver information to 
the public. While agencies have made some progress in establishing finding aids such as 
the Government Information Locator Service (GILS) to assist in locating information, as 
yet no systematic effort has been devoted to measuring the overall performance of 
websites in successfully achieving adequate levels of public access to government 
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information. 



Some agencies maintain a range of statistics describing web services while others have 
undertaken only minimal or no data collection and analysis effort; some have devoted 
substantial resources to “one-stop shopping” for information; many have developed 
“frequently asked questions” to assist visitors to agency websites. Most agencies already 
use weblog statistics and other software-based measures to examine aspects of their 
websites' performance. But agencies still need a flexible approach that goes beyond web 
statistics such as transaction logs to offer a variety of techniques by which agencies can 
determine whether their websites are successfully achieving the information 
dissemination missions for which they are intended. 



1.1 STUDY PURPOSE AND REPORT STRUCTURE 

The overall purposes of the project are to develop performance measures that will assist 
agencies to assess the quality and usefulness of their websites and to improve public 
access services of those websites. More specifically, the project had the following 
objectives: 

• Examine the professional literature, as well as government agency and 
commercial experience, to identify and make use of measures already developed 
by others. 

• Identify those aspects of agency websites that are most important for effective 
delivery of public information. 

• Develop a set of objective performance measures for determining the degree to 
which an agency website effectively provides public access to agency information 
resources. 

• Test and refine these measures in selected agencies. 

• Propose a concise set of procedures that describes how these performance 
measures can be further developed and applied by agencies. 

Limited time and resources did not allow the investigators to test and refine the measures 
as much as might have been desired. Such efforts can be continued after this study. 
Ultimately, the investigators hope that both public user groups and agency officials will 
be able to determine the degree to which a given agency website meets these performance 
measures. 

The study provides an overview of evaluation techniques and performance measures used 
in three federal government agencies (see Chapter 3). Based on these summaries, 
selected sources from the literature, and the investigators’ experience, the report then 
suggests key federal website performance measures that can be developed and used in 
federal agencies at this time (see Chapter 4). 

The report is structured as follows. This first chapter provides an introduction to key 
concepts and literature related to evaluation and performance assessment. The second 
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chapter reviews federal policy impacting federal agency web site evaluation. The third 
chapter summarizes performance measures used at three federal agencies. The fourth 
chapter suggests federal agency performance measures for the future. The report 
concludes with a chapter describing key issues yet to be resolved and offers some 
possible next steps in developing performance measures for federal websites. 

1.2 OVERVIEW OF STUDY METHOD 

The study employed a multi-method approach to data collection and analysis. Given the 
relatively short time line for the study and the limited resources available the following 
approaches were employed: 

• Literature Review. The study reviewed selected literature and reports from both 
inside and outside the government. While there is some useful background 
information (see below) there is limited knowledge and information about 
developing and computing a coherent set of performance measures for 
government websites. 

• Group Interviews with Participating Agency Representatives. The 

investigators met (1) individually with representatives of each participating 
agency, (2) the participants as a group, and (3) conducted a number of individual 
follow-up discussions with representatives of the three agencies about activities 
and procedures for assessment in their agency. 

• Best Practices. Each of the three participating agencies provided the 
investigators with (1) descriptions of their assessment techniques, and (2) 
documentation and reports related to these assessment techniques; these materials 
were reviewed and analyzed by the investigators. 

• Policy Analysis. The investigators reviewed existing federal policies, laws, and 
guidelines to produce an overview of policies affecting the development and 
management of federal websites; this compendium identifies a number of 
ambiguities and issues that will require additional assessment. 

• Survey. The investigators conducted a survey at the FedWeb 2000 conference 
held in March, 2000 at Bethesda, MD. This survey provided a snapshot of current 
practices and key issues of concern to federal officials regarding the assessment of 
federal websites. 

e Participant Review. Officials from the three participating agencies had a 

number of opportunities to review preliminary findings and reports developed by 
the investigators. 

The combination of these approaches provided the investigators with a unique 
perspective on current practices, key issues, and possible assessment techniques and 
performance measures. The approach demonstrated the importance of recognizing the 
existing policy environment that affects agency website development and management 
(see Chapter 4). The approach also resulted in a better understanding of the problems and 
issues that agencies confront in attempting to conduct ongoing website evaluation and 
assessment (see Chapter 5). 
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1.3 WHAT IS EVALUATION? 



Website evaluation can be defined as the use of research or investigative procedures to 
systematically investigate the effectiveness of a web based information system on an 
ongoing basis. Evaluation is used in conjunction with a number of activities and has a 
number of impacts on an organization. 



1.3.1. Planning, Goal Setting and Determining the Degree to Which the IS 
Organization is Meeting Set Goals 

Evaluation plays a key role in organizational planning and goal 
setting. This is known as “formative” evaluation. In contrast, 
“summative” evaluation determines the degree to which the 
organization is meeting set goals. Figure 1 illustrates this dual 
role. On the left side of the diagram, information discovered as 
part of the evaluation process feeds back into goal setting and 
planning. Ongoing evaluation is a vital source of information for 
agencies’ planning processes. For example, an evaluation of 
current website user satisfaction may reveal usability issues with 
the current page design or information architecture. Planners may 
choose to change or modify goals based upon newly discovered 
problems or the achievement of previously set goals. 




Figure 1.1: Formative and Summative Evaluation 
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On the right side of the diagram, evaluation is used to determine the degree to which the 
organization has met stated goals. Developing goals and objectives with no follow-up 
effort to determine how well those objectives were actually accomplished significantly 
reduces the overall value of both planning and the use of assessment techniques. Based 
on the previous evaluation, if the organization had created a goal to improve site 
usability, they would then use evaluation to determine the degree to which the site’s 
usability had improved. 
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Both formative and summative evaluation efforts are important - although most 
organizations tend to concentrate on summative approaches. But for monitoring and 
ongoing improvement of services, formative evaluation (intended to improve, not prove) 
is essential. 

1.3.2. Decision-making and Resource Allocation 

Often managers find themselves in problematic situations with multiple possible 
solutions. It is difficult to know, up front, which solution will best meet overall goals. 
Consider the situation where a web manager’s overall goal is to increase the user’s ability 
to find product and service information. Designers might suggest a number of possible 
web page designs. Which design will best increase usability? Evaluation of the possible 
page designs will provide data to help answer the question. 

Further, in an environment of restricted resources, organizations often cannot provide 
resources to all programs or projects that request funding. Decision makers must choose 
projects or programs to fund. Evaluation can provide data to assist in the decision 
making process and justify requests for new resources. 

1.3.3. Determine the Effectiveness of a Previous Decision 

Similarly, organizations can use evaluation to determine the effectiveness of a previous 
decision. Did the funded program or project live up to its promises? To what extent did 
the funded program or project meet its stated goals? To what extent did the decision 
successfully contribute to the organization’s goals? When agencies fail to ask these types 
of questions they run the risk of maintaining programs and services that do not work well 
and are not meeting user needs. 



1.3.4. Determine the Degree to which the System/Service Adds Value to the 
Organization 

Evaluation can answer specific organizational questions about the degree to which a 
system adds value to an organization. For instance, evaluation could show how a system 
reduced overall costs or to determine if printing costs decline as on-line publication of 
information increases. 

Agencies are only beginning to explore how web based services add value or enhance 
traditional services. But, agencies now see that being able to provide user services on a 7 
days a week, 24 hours a day basis is a significant change in service provision. This 
“value add,” however, may result in other impacts such as an increased need for real time 
user support. 

1.3.5. Provide Trend Data to Assess Change Over Time 
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Collection and comparison of assessment results over time can show trend data for 
federal web sites. Trend results may show changes more effectively than individual 
assessment results if the results change gradually over time and imply a continued trend 
in a certain direction. For instance, if results were being used to justify more budget 
dollars, trend data showing reduced costs over time would both better show the impact of 
the service and more effectively convince decision makers that costs will continue to fall. 

Trend data is especially important for formative evaluation efforts as it provides a basis 
by which changes in the use of a services can be identified early on. In log analysis 
techniques, a change in accessing the website via a second level page rather than the 
homepage has significant implications for content development. 

1.3.6. Contribute to Continuous Improvement Efforts and Benchmarking 

Evaluation can serve as a first step in benchmarking. In benchmarking, organizations 
identify best-practice performance. They then use that performance as a goal, investigate 
the factors that led to the performance, and then try to replicate that level of performance. 
Evaluation provides a base level reading of organizational performance on benchmark 
measures and can also show longitudinal progress toward the goal. 

1.3.7. Identify Problems and Possible Solutions 

One impact of evaluation is the identification of previously undetected problems. 
Evaluation can also indicate the extent or scope of a problem. Once such problems are 
identified and described, a plan to address the problem can be developed. Identification 
of unanticipated problems is a powerful tool that takes on increased important in a web 
environment since oftentimes technical assessment (as opposed to user assessment) may 
be the only type of assessment being conducted. 

1.3.8. Empower Organizational Actors to Seek and Enact Solutions 

Another beneficial side effect of evaluation is that it can lead to increased employee 
involvement and action. As part of the evaluation process, employees identify problems 
and potential solutions. Employees then can act on the problems, develop solutions and 
improve web-based services directly and immediately. Seeing changes and 
improvements to services can improve morale and increase confidence to tackle further 
problems. Further, changes to services may improve work processes and reduce 
inefficiency, thereby reducing work stress. 

1.3.9. Develop Accountability 

Evaluation may also lead to increased accountability as employees within the 
organization identify and take responsibility for problems and solutions. Further, the goal 
setting process in formative evaluation has the positive impact of increasing employees’ 
understanding of organizational goals and expectations. This better understanding may 
lead to increased accountability in meeting the goals and expectations. 
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1.3.10. Organizational Learning 

Evaluation is also an opportunity for organizations to learn about themselves and the 
diversity of needs, goals and perceptions within the organization. First, evaluation tells 
the organization about its own strengths and weaknesses. Second, it provides a snapshot 
of current organizational infrastructure and resources. Third, evaluation can illuminate 
differences between groups within the organization; how the groups may have different 
goals, different interpretations of problems, and differing needs. Finally, the evaluation 
process encourages these value judgments regarding appropriate levels or quality of 
services to be made explicit. 

These value judgments may provoke debate that educates members about the potential 
impact of particular measurements on the organization, the costs and benefits of choosing 
a particular goal, and the variety of needs and goals of different groups within the 
organization. 

1.4. PERFORMANCE MEASURES 

Performance measures are the tools that enable evaluation. As a doctor needs to monitor 
blood pressure and heartbeat to perform a check-up, evaluators need to look at 
performance measures to conduct an evaluation. They are an essential means to assess 
agency services and accomplishment of mission. 

Performance measures typically provide measurements of the following: 

• Inputs and Efficiency. The use of resources in providing or accessing web 
services (e.g. staff hours required for site maintenance, technical training or 
consulting services required for systems integration). 

• Effectiveness. How well the networked information service met the objectives of 
the provider, user or customer (e.g. success rate of identifying and accessing the 
information needed by the user). 

• Outputs. Indicators of the products or services resulting from the use of those 
resources (e.g. number of agency documents available online, usage levels for a 
given e-service). 

• Extensiveness. How much of a service the network provides (e.g. number of 
visits to a web site per week, how many document downloads per week.) 

• Service Quality. How well a service or activity is done (e.g., percentage of 
transactions which result in an order or a document download). 

• Impact. How a service made a difference in some other activity or situation (e.g. 
the degree to which network users saved time, resolved a decision or identified 
new and innovative applications/services). 

• Usefulness. The degree to which the services are useful or appropriate for 
individual users (e.g. percentage of services of interest to different categories of 
users). 

• Adoption. The extent to which institutions or users integrate and adopt web 
resources in organizational or individual activities (e.g. problem solving, using 
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online forms instead of print, etc.). 

These categories can become blurred and are not necessarily mutually exclusive. In 
addition, evaluation and performance measures reflect value judgments on the part of 
evaluators regarding the adequacy, appropriateness, and success of a particular service or 
activity. Use of performance measures can lead to beneficial organizational changes. 

For example, a tendency of users stick to use of printed forms instead of online forms can 
be an indicator that the organization needs some targeted business process reengineering. 

1.5. OVERVIEW OF WEB EVALUATION 

This section sets the context for federal web site evaluation, explaining its relation to 
broader efforts to evaluate web information systems (web IS) and information systems 
(IS) in general. One can see federal agency web system evaluation as a special subset of 
web IS evaluation that in turn is a subset of general IS evaluation. This section first 
provides a brief overview of IS evaluation in general. It then provides a select overview 
of modem web IS evaluation research. It finishes with an overview of recent federal web 
evaluation efforts. 

1.5.1. IS Evaluation 

IS evaluation has become an increasingly important topic within the competitive U.S. 
business environment. Several factors have contributed to evaluation’s growing 
importance. First, IS projects historically have had low success rates; some researchers 
have suggested they are as low as 30-40 percent (Willcocks and Lester, 1993). Tom 
Peters believes that 85 percent of companies trying to restructure around e-commerce will 
“blow it.” He goes on to suggest that the move to e-commerce has more to do with 
relationships and organization than with IT. (Infoworld, September 25, 2000, p. 200). 

This increases pressure on managers to both justify their projects and show how their 
projects can and will succeed. Second, vendors inundate mangers with a maelstrom of 
hype surrounding new products and IT trends. 

Managers need evaluation tools to help them determine the actual usefulness of these 
products and trends for their organizations. Third, while organizations’ budgets have 
generally increased allocations for IT, downsizing and streamlining demands require IT 
managers to show how increased IT spending is adding value to the organization. For 
examples of and resources about general IS evaluation guides see: 

• The National Research Council report More Than Screen Deep: Toward Every- 
Citizen Interfaces to the Nation 's Information Infrastructure, 

http : //www. nap . edu/readingroom/books/screen 

• Assessing the Academic Networked Environment: Strategies and Options by 
Charles McClure and Cynthia Lopata, available through the Coalitation on 
Networked Information, http://www.cni.org 

• “Managing Information Systems: A Practical Self Assessment Tool” (Preview 
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version) by the Information Technology Review Board, http://www.itrib.gov 

• Performance-Based Management: Eight Steps to Developing and Using IT 
Measures Effectively” by the GSA Office of Government wide Policy, 
http://www.itpolicy.gsa.gov/mkm/pathways/evt8step.html 

• Performance Pathways” part of the IT Ramp website of the GSA Office of 
Information Technology http://itpolicy.gsa.gov 

Readers should take note that the many resources developed for general IS evaluation can 
be adapted for use in web site evaluation. 

1.5.2. Web IS Evaluation 

The last five years has seen a surge of interest in website evaluation. One result has been 
the publication of plethora of web “do-it-yourself’ books that include advice on both 
design and evaluation (for example see Nielsen, 2000; Jacobson, 1999). At the same 
time researchers from the business, education and library science fields have sought to 
evaluate web sites based on the many criteria including: 

• Interface design (Kopak and Cherry, 1998; Vanhouse, Butler, Ogle, Schiff, 1996) 

• Usability (Benbunan-Fich, 1999) 

• Comparison to peer organizations - benchmarking (Johnson and Misic, 1999) 

• Fit with theoretical models (eg. Marketing model: von Dran; Zhang and Small, 
1999; motivational model: Zhang and von Dran, 2000) 

• Web site strategy (Simeon, 1999; Auger 1997) 

• Information quality (McMurdo, 1998) 

• Hypertext Structure (Bauer and Scharl, 2000) 

Web site evaluation has also become a popular topic within the trade press (e.g. Dugan, 
2000). A significant amount of web evaluation effort has been put into log analysis 
techniques (Bertot, McClure and Rubin, 1997) and use of specific log analysis software 
such as WebTrends (http://webtrends.com) . Additional information on the use of log 
analysis tools and software can be found at Yahoo! Under computers and the Internet: 

Log Analysis tools at 

http://www.yahoo.com/Computers and Intemet/Software/Intemet/World Wide Web/Se 
rvers/Log-Analysis Tools . 

Once again, readers should note that information on general website evaluation is 
applicable to the federal web environment with certain key allowances made for design 
restraints imposed by regulation or statute. 

1.5.3. Federal Website Evaluation 

Federal website evaluation has been ongoing since the inception of federal websites. One 
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early landmark was the World Wide Web Federal Consortium 1 publication of suggested 
guidelines for federal website development (draft 1996). The guidelines have been 
periodically updated in recent years. 2 Many federal agencies conduct periodic 
evaluations to maintain and enhance the quality of their sites. See, for example, “Love 
That Site, Love That Agency” ( Government Computer News, 1999) for a description of 
an evaluation of the Defenselink website. Recently published guidelines include the 
Federal Webmaster Forum Website Privacy Guidelines. 3 

There has also been a substantial and increasingly sophisticated academic evaluation 
research stream. Current web evaluation research has looked at federal websites in terms 
of a variety of evaluation criteria including information content and ease of use (e.g. 
Eschenfelder et al., 1997; McClure and Wyman, 1997; Fieri and Marchionini, 1997) and 
compliance with federal records guidelines (McClure and Sprehe, 1999). 

Further, some studies have looked at specific aspects of websites. For instance, Hert 
(1998) evaluated web site finding aids and Moen and McClure (1997) examined the 
government information locator service (GILS). Other evaluation efforts have taken a 
more holistic approach. For instance, Hert, Eschenfelder and McClure (2000) included a 
usability, management, technical and policy analysis. Finally, these studies have been 
varied in methodologies, with some relying on mainly one method (e.g. log file analysis 
Redalen & Miller (2000) and Bertot et al. (1997)) while others have taken a multi-method 
approach (e.g. Hert et al., 2000). 

1.6. IMPORTANCE OF WEBSITE EVALUATION 

In summary, web site evaluation is an important part of the ongoing management and 
enhancement of federal agency websites. Evaluation, whether as part of a formal 
planning process or as a stand-alone activity, can provide managers with key information 
that can aid decision making. Although web sites are a relatively new medium, IS 
evaluation has a long history, providing tools and lessons learned that federal web 
managers can apply to their web evaluation efforts. Current academic research on federal 
agency web evaluation also provides strategies and tools that agencies can adapt for their 
efforts. 

Interest in evaluation of web based services continues to increase, although, few 
comprehensive approaches assess federal websites on an ongoing basis. As agencies 
continue to be pressed to provide additional web based services with limited resources 
and as implementation of the Government Performance and Results Act presses forward, 
ongoing evaluation and use of performance measures is likely to take on increased 
importance. 



1 The original guidelines is still available at http://www.dtic.mil/staffycthomps/guidelines (last visited 
August 2000). 

2 The current version of the guidelines, dated July 1999, can be found at 

http://diggov.org/library/current/general/ www.oip.usdog.gov A oa A fedwebguide A execsum.htm . (Visited 
August 2000) 

3 See the Federal Webmaster Forum homepage at http://itpolicy.gsa.gov/mke/fedwebm/fedwebm.html for 
current Forum website privacy guidelines. 
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CHAPTER 2 



FEDERAL POLICIES AFFECTING WEBSITE DEVELOPMENT 

AND MANAGEMENT 



2.0 INTRODUCTION 

Federal information policies establish the legal and procedural framework in which 
government information and services are made available to the public. "An information 
policy instrument is a written law, guideline, regulation, or other official statement that 
describes how information will be collected, managed, protected, accessed, disseminated, 
and used" (Kent 1999, 307). The purpose of this chapter is to provide an introductory 
review of selected U.S. Federal information policy instruments that affect the 
development and management of Federal websites. While specific references to the 
Internet or the World Wide Web may not appear in a given policy instrument, this does 
not necessarily mean the policy has no application to Federal websites. These 
instruments are directed toward Federal government procedures and operations in the 
context of Federal information and/or information technology, and thus would apply to 
Federal websites unless special rules require otherwise. 

This listing is not comprehensive, but rather offers a general sense of the range of 
existing Federal policies that should be considered in the development, management, and 
evaluation of websites. Where possible, citations to Web-based sources of information 
regarding a particular policy are provided, as are selected excerpts of the policy that may 
be of interest to Federal Web designers, managers, and evaluators. The selection of 
policies could be extended much more broadly. For example, the chapter leaves out the 
area of electronic commerce. Federal agencies do use their websites for selling goods 
and services, and electronic buying of goods and services is a major thrust in the Federal 
procurement community. Readers who wish to explore the area of Federal e-commerce 
are referred to http://www.ecommerce.gov . 

The policy instruments are organized into a schematic outline, moving from general 
government policy to general information policy and then to specific information 
policies. Within each heading occurs a listing of statutory policies and implementing 
policy guidance. It is within this context that Federal agencies develop and manage their 
websites for providing government information and services, which must also meet the 
agencies’ mission statements, objectives, and goals. 

Issues that Federal agencies need to consider in developing websites include information 
security, information privacy, information access, electronic records management, and 
intellectual property. Information security instruments concern risks to the ongoing 
operation of government computer systems, their integrity, and the protection of 
classified or confidential materials they contain. Information privacy instruments seek to 
protect personal information that may be collected from agency website users. Internet 
access policy instruments are concerned with ensuring the equitable access for U.S. 
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citizens to electronic information contained on Federal government websites. Electronic 
records management policy instruments concern issues regarding the creation, 
maintenance, use, and disposal of Federal records. Intellectual property policy 
instruments include a wide variety of ownership rights in intangible products, such as 
copyrights, patents, trademarks, and trade secrets. 

Federal information policy and agency website development occur in a dynamic 
environment. Technological changes impacting on established information policies or 
creation of new ones occur rapidly, but Federal agencies often must adjust their 
operations almost immediately. Policy tends to follow technology and practice. 
Sometimes the lag between policy and practice can be great, so that agencies must craft 
their own policies to rationalize practices before Congress enacts new laws. 

2.1 GENERAL GOVERNMENT POLICY 

2.1.1. Government Performance and Results: Performance Plans and 
Measures 

2. 1.1.1. Statute : Government Performance and Results Act of 1993. 
(Public Law 103-62). 

Available at: < http ://thomas . loc . go v/cgi-bin/bdq uery/ 
z?d 1 03 : SN00020: ITOM.bss/dl 03query.html >. 

Application to Federal Information Technology and Web 
Management: 

The Government Performance and Results Act of 1993 (GPRA) is 
intended to help reduce waste and inefficiency in Federal programs and 
Federal agency operations. It is also designed to assist Congress and the 
Executive in their oversight, legislative, and administrative tasks related to 
authorizing, appropriating, and implementing Federal services. GPRA 
mandates the adoption of a strategic and annual planning process, which is 
tied to budget and authorization cycles and will be based on established 
and measurable performance indicators for every program. Although this 
act was made law prior to the expansion and wide public use of the 
Internet, particularly the Web, the mandate for the development of 
performance indicators tied to annual budgeting and strategic planning 
applies to services offered in an electronic environment. Performance 
indicators for Federal websites consist of measures that permit an agency 
to demonstrate whether its websites are or are not meeting the 
performance goals set forth for the sites. Performance measures could 
include things such as log transaction files and impact measurements. 

Excerpt of Original Text from Policy Instrument: 

Section 3. Strategic Planning 
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Chapter 3 of title 5, United States Code, is amended by adding after 
section 305 the following new section: 

Sec. 306. Strategic plans 

(а) No later than September 30, 1997, the head of each agency shall 
submit to the Director of the Office of Management and Budget and to the 

Congress a strategic plan for program activities. Such plan shall contain— 
* * * 

(3) a description of how the goals and objectives are to be achieved, 
including a description of the operational processes, skills and technology, 
and the human, capital, information, and other resources required to meet 

those goals and objectives; 

* * * 

(б) a description of the program evaluations used in establishing or 
revising general goals and objectives, with a schedule for future program 

evaluations. 

* * * 



Section 4. Annual Performance Plans and Reports 
* * * 

(b) PERFORMANCE PLANS AND REPORTS- Chapter 1 1 of title 31, 
United States Code, is amended by adding after section 1 1 14 the following 
new sections: 

Sec. 1115. Performance plans 

(а) In carrying out the provisions of section 1 105(a)(29), the Director of 
the Office of Management and Budget shall require each agency to 
prepare an annual performance plan covering each program activity set 
forth in the budget of such agency. Such plan shall— 

(1) establish performance goals to define the level of performance to be 
achieved by a program activity; 

(2) express such goals in an objective, quantifiable, and measurable form 
unless authorized to be in an alternative form under subsection (b); 

(3) briefly describe the operational processes, skills and technology, and 
the human, capital, information, or other resources required to meet the 
performance goals; 

(4) establish performance indicators to be used in measuring or assessing 
the relevant outputs, service levels, and outcomes of each program 
activity; 

(5) provide a basis for comparing actual program results with the 
established performance goals; and 

(б) describe the means to be used to verify and validate measured values. 

2.1. 1.2. Implementing Guidance: OMB Circular No. A-ll, Part 

II, Preparation and Submission of Strategic Plans, Annual 
Performance Plans, and Annual Program Performance 
Reports 
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Available at <http://www.whitehouse.gov/omb/circulars/al l/99toc.html>. 

Application to Federal Information Technology and Web 
Management: 

OMB Circular No. A-ll, issued annually, instructs federal agencies how 
they are to submit their budget requests for two years hence; that is, in 
fiscal year (FY) 2000 agencies submit budget requests for FY 2002. Part 
II was added to Circular A-ll after the passage of the Government 
Performance and Results Act. It sets forth detailed procedural guidance 
for formulating the agencies’ annual performance plans and describes 
what agencies are to include in their annual program performance report. 
Included in Part II is the requirement for program performance goals and 
the enumeration of the performance measures agencies will use to assess 
their relative success or failure in achieving performance goals. Part II is 
the basic Federal policy guidance regarding performance measures in 
general, as applied to all Executive Branch programs, and has applicability 
to Federal websites. 

2.1.2. Customer Service Standards: Setting Customer Service Standards. 
(Executive Order 12862). 1993. 

Available at: < http://www.pub.whitehouse.gov/uri-res/ 

I2R?um:pdi://oma.eop.gov.us/1993/9/14/3.text.2 >. 

Application to Federal Information Technology and Web Management: 

Although this Executive order does not specifically address Customer 
Service Standards in an electronic environment, Federal agency website 
development teams still need to identify their customers, their customers’ 
needs, and set standards and benchmarks. 

Excerpt of Original Text from Policy Instrument: 

Sec. 1. Customer Service Standards. * * * 

All executive departments and agencies (hereinafter 
referred to collectively as "agency" or "agencies") that provide 
significant services directly to the public shall provide those 
services in a manner that seeks to meet the customer service 
standard established herein and shall take the following 
actions: 

(a) 

identify the customers who are, or should be, served by the 
agency; 

(b) survey customers to determine the kind and quality of 
services they want and their level of satisfaction with 
existing services; 

(c) post service standards and measure results against them; 
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(d) benchmark customer service performance against the best in 
business; 

(e) survey front-line employees on barriers to, and ideas for, 
matching the best in 

business; 

(f) provide customers with choices in both the sources of service 
and the means of 

delivery; 

(g) make information, services, and complaint systems easily 
accessible; and 

(h) provide means to address customer complaints. 

Provisions of the Executive Order also require agencies 
to have a “Customer Service Plan,” to develop assessment 
techniques to gauge the success of the plans, and to report 
on the degree to which the plan is being accomplished. 

2.1.3. Accessible Electronic and Information Technology 

Rehabilitation Act of 1973 (Pub. L. 93-112), section 508 (added by the 
Rehabilitation Act Amendments of 1986, Pub. L. 99-506, section 
603(a), codified as amended at 29 U.S.C. § 794d). 1998. 

Available at: < http://frwebgate.access.gpo.gov/cgi- 
bin/getdoc.cgi?dbname=l994 uscode suppl 4&docid=29usc794d > 



Application to Federal Information Technology and Web 
Management: 

Section 508 requires that Federal agencies’ electronic and information 
technology be accessible to people with disabilities, including employees 
and members of the public. It also establishes requirements for any 
electronic and information technology developed, maintained, procured, or 
used by the Federal government. The Attorney General must determine 
and communicate what information is necessary from Federal agencies to 
conduct evaluations of their current electronic and information technology 
systems’ accessibility. Agencies must then evaluate the accessibility of 
their IT to individuals with disabilities. In addition, by February 7, 2000, 
the Architectural and Transportation Barriers Compliance Board is 
required to issue standards that define which electronic and information 
technology is covered and describe what is meant by “accessible 
technology” by setting forth technical and functional performance criteria. 
The deadline for issuing of the Standards has been extended to August 7, 
2000. Federal agency website development teams can find additional 
information at the Center for IT Accommodation (CITA) to aid in their 
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design of websites accessible to people with disabilities. CITA is 
available at <http://www.itpolicy.gsa. go v/cita/index ,htm >. 

Excerpt of Original Text from Policy Instrument: 

(a) Requirements for Federal departments and agencies 

(1) Accessibility 

(A) Development, procurement, maintenance, or use of 
electronic and information technology 

When developing, procuring, maintaining, or using 
electronic and information technology, each Federal department or 
agency, including the United States Postal Service, shall ensure, 
unless an undue burden would be imposed on the department or 
agency, that the electronic and information technology allows, 
regardless of the type of medium of the technology — 

(i) individuals with disabilities who are Federal employees 
to have access to and use of information and data that is 
comparable to the access to and use of the information and data by 
Federal employees who are not individuals with disabilities; and 

(ii) individuals with disabilities who are members of the 
public seeking information or services from a Federal department 
or agency to have access to and use of information and data that is 
comparable to the access to and use of the information and data by 
such members of the public who are not individuals with 

disabilities. 

* * * 

(c) Agency evaluations 

Not later than 6 months after the date of enactment of the 
Rehabilitation Act Amendments of 1998, the head of each Federal 
department or agency shall evaluate the extent to which the 
electronic and information technology of the department or agency 
is accessible to and usable by individuals with disabilities 
described in subsection (a)(1), compared to the access to and use of 
the technology by individuals described in such subsection who are 
not individuals with disabilities, and submit a report containing the 
evaluation to the Attorney General. 

2.1.4. President’s Memorandum on Electronic Government. December 
17,1999. 

Available at: 

< http://www.pub.whitehouse.gov/uri- 

res/I2R?um:pdi://oma.eop.gov.us/1999/12/20/5.text.l > 

Application to Federal Information Technology and Web 
Management: 
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President Clinton announced twelve actions that Federal agencies 
can take in conjunction with private industry to provide American 
citizens with improved access to government services and information. 
The President desired Federal agencies to provide standardized access 
to government information and services. People should be able to find 
needed government information without needing to know what agency 
is responsible for disseminating the information. People should also 
be confident that that their communication with government is secure 
and their privacy protected. 

Excerpt of Original Text from Policy Instrument: 

2. The heads of executive departments and agencies 

(agencies) shall, to the maximum extent possible, make 
available online, by December 2000, the forms needed for 
the top 500 Government services used by the public. Under 
the Government Paperwork Elimination Act, where 
appropriate, by October 2003, transactions with the Federal 
Government should be available online for online 
processing of services. To achieve this goal, the Director of 
the Office of Management and Budget shall oversee agency 
development of responsible strategies to make transactions 
available online. 

4. The heads of agencies shall continue to build good privacy 
practices into their web sites by posting privacy policies as 
directed by the Director of the Office of Management and 
Budget and by adopting and implementing information 
policies to protect children's information on web sites that 
are directed at children. 

5. The head of each agency shall permit greater access to its 
officials by creating a public electronic mail address 
through which citizens can contact the agency with 
questions, comments, or concerns. The heads of each 
agency shall also provide disability access on Federal web 
sites. 

7. The Secretaries of Health and Human Services, Education, 
Veterans Affairs, and Agriculture, the Commissioner of 
Social Security, and the Director of the Federal Emergency 
Management Agency, working closely with other Federal 
agencies that provide benefit assistance to citizens, shall 
make a broad range of benefits and services available 
though private and secure electronic use of the Internet. 



McClure, Sprehe & Eschenfelder 



2-7 



Federal Website Performance Measures 



9. The heads of agencies shall develop a strategy for 

upgrading their respective agency's capacity for using the 
Internet to become more open, efficient, and responsive, 
and to more effectively carry out the agency's mission. At a 
minimum, this strategy should involve: (a) expanded 
training of Federal employees, including employees with 
policy and senior management responsibility; (b) 
identification and adoption of "best practices" implemented 
by leading public and private sector organizations; (c) 
recognition for Federal employees who suggest new and 
innovative agency applications of the Internet; (d) 
partnerships with the research community for 
experimentation with advanced applications; and (e) 
mechanisms for collecting input from the agency's 
stakeholders regarding agency use of the Internet. 



2.2. FEDERAL INFORMATION POLICY 

2.2.1. U.S. Information Infrastructure Task Force (IITF). 1993 The National 
Information Infrastructure: Agenda for Action. 

Available at: < http://metalab.unc.edu/nii/toc.html >. 

Application to Federal Information Technology and Web 
Management: 

This document has served as a blueprint for President Clinton’s National Information 
Infrastructure Initiative. The document outlined the President’s map for 
developing a National Information Infrastructure (Nil) listing goals and 
the necessary actions to accomplish the goals. Goal nine called for an 
increase in access to government information and to improve government 
procurement. This goal called for Federal agencies in conjunction with 
state and local governments to use the Nil to expand the information 
available to the public, and make that information so it can be accessed 
easily and equitably. 

Excerpt of Original Text from Policy Instrument: 

9. Provide Access to Government Information and Improve 
Government Procurement 

Thomas Jefferson said that information is the currency of 
democracy. Federal agencies are among the most prolific 
collectors and generators of information that is useful and valuable 
to citizens and business. Improvement of the nation's information 
infrastructure provides a tremendous opportunity to improve the 
delivery of government information to the taxpayers who paid for 
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its collection; to provide it equitably, at a fair price, as efficiently 
as possible. 

The Federal government is improving every step of the 
process of information collection, manipulation, and 
dissemination. The Administration is funding research programs 
that will improve the software used for browsing, searching, 
describing, organizing, and managing information. But it is 
committed as well to applying those tools to the distribution of 
information that can be useful to the public in their various roles as 
teachers, researchers, businesspeople, consumers, etc. 

Action: Improve the accessibility of government 
information. IITF working groups will carefully consider the 
problems associated with making government information broadly 
accessible to the public electronically. Additionally, several inter- 
agency efforts have been started to ensure that the right 
information is stored and available. Finally, to help the public find 
government information, an inter-agency project has been formed 
to develop a virtual card catalogue that will indicate the availability 
of government information in whatever form it takes. 

Action: Enhance citizen access to government information. 

In June 1993, OMB prescribed new polices pertaining to the 
acquisition, use, and distribution of government information by 
Federal agencies. Among other things, the policies mandate that, in 
distributing information to the public, Federal agencies should 
recoup only those costs associated with the dissemination of that 
information, not with its creation or collection. Moreover, a 
number of inter-agency efforts are under way to afford greater 
public access to government information. One project seeks to turn 
thousands of local and field offices of various Federal agencies 
into Interactive Citizen Participation Centers, at which citizens can 
communicate with the public affairs departments of all Federal 
agencies. 

2.2.2 Privacy and the National Information Infrastructure: Principles for 

Providing and Using Personal Information. Privacy Working Group. 
Information Policy Committee. Information Infrastructure Task 
Force. (June 6, 1995.) 

Available at: < http://www.iitf.nist.gov/ipc/ipc/ipc- 
pubs/niiprivprin fmal.html >. 

Application to Federal Information Technology and Web 
Management: 

The Privacy Working Group (PWG), part of the Information Infrastructure Task Force 
(IITF) was formed to develop guidelines or principles for providing and 
using personal information in the electronic environment or within the 
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National Information Infrastructure (Nil). Personal information refers to 
information such as identifying information, social security number, name, 
address, and phone number; financial records or statements; and health 
records. The principles were designed to provide guidelines to personal 
information users and personal information providers. These principles are 
not legally binding, and only serve to guide. In developing the standards, 
PWG generally followed international standards such as the OECD 
guidelines written in 1 980. 

Federal website developers and administrators need to be concerned with 

• Acquisition Principle: only collect and keep information that supports 
current or planned activities, and assess the impact on privacy in 
collecting, disclosing, or using a person's personal information. 

• Protection Principle: use appropriate technical and managerial 
controls to protect the integrity, confidentiality, and quality of the 
personal information. 

• Notice Principle: personal information providers should be informed 
by the website as to why information collection is occurring; how the 
information is to be used; what steps are taken to protect the 
information; consequences of providing or withholding information; 
any rights of redress. What happens if information loses integrity, or 
is leaked. 

• Fairness principle: Information should be used in the manner in which 
the user was told it would be used. 

• Education principle: Information users should educate themselves and 
the public about how information privacy can be maintained. 

2.2.3. COPYRIGHT ACT 

2.2.3.1 Statute ; Digital Millennium Copyright Act. (Public Law 105- 
304, Title 1, Sec. 103(a), codified at 17 U.S.C. §§ 1201-1205). 
1998. 

Available at: < http://frwebgate.access.gpo.gov/cgi- 
bin./getdoc.cgi?dbname=1994 uscode suppl 4&docid=17uscl201 
> 

Application to Federal Information Technology and Web 
Management: 

This law amended Title 1 7 of the U.S. Code to implement the World 
Intellectual Property Organization Copyright Treaty and Performance and 
Phonograms Treaty. The Digital Millennium Copyright Act (DMCA) 
incorporated into Title 17 provisions preventing the circumvention of 
technological protections of works. The DMCA also added provisions 
prohibiting the removal or alteration of “copyright management 
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information.” As such, the DMCA tailors protection to holders of 
copyright in electronic media. It prohibits the circumvention of 
technological measures that control access to protected works and/or 
manufacturing or trafficking in technology designed to circumvent 
measures that control access to, or protect rights of copyright owners in, 
such works. The act provides exemptions for nonprofit libraries, archives, 
or educational institutions that gain access to a commercially exploited 
copyrighted work solely to make a good faith determination of whether to 
acquire such work, subject to certain restrictions; for purposes of 
achieving interoperability of computer programs; and for authorized 
investigative, protective, information security, or intelligence activities of 
the United States, a state, or political subdivision of a state. 

2.2.3.2 Implementing Guidance: The Copyright Office completed a 
statutorily mandated study on the Digital Millennium Copyright Act in 
December 1998. The study is available at 
< http://www.loc.gov/copyright/legislation/dmca.pdf >. 

2.2.4 RIGHTS OF ACCESS TO INFORMATION 
2.2.4.1 FREEDOM OF INFORMATION 

2.2.4.I.I. Statute : Freedom of Information Act. (Public Law 89-487, 
codified as amended, 5 U.S.C. § 552). 

Available at: < http://frwebgate.access.gpo.gov/cgi- 
bin/getdoc.cgi?dbname=1994 uscode suppl 4&docid=5usc552 > 

Application to Federal Information Technology and Web 
Management: 

The purpose of the Freedom of Information Act (“FOIA”) is to allow 
public access to agency records, thereby promoting an informed citizenry. 
Unless exempted under section 552a, agency records are subject to proper 
request. EFOIA eliminated doubt that information stored on computers is 
a “record” within the definition of the statute. Because digital 
information, including information posted by a Federal agency on its 
website, falls under the statutory definition of a record, Federal agencies 
are responsible for providing information in this format just as they are for 
providing print materials. 

Certain government bodies are exempted from the requirements of 
FOIA because they fall outside the definition of “agency” set forth in the 
statute, and the law contains nine exemptions for various classes of 
records. As noted in the section dealing with the Privacy Act, certain 
types of information are protected from public disclosure and 
consequently their disclosure is not required under FOIA. Each section is 
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defined in terms of the other: records required under FOIA include 
everything not protected under the Privacy Act. 

2.2.4.1.2. Statute : Electronic Freedom of Information Act 
Amendments of 1996. (Public Law 104-231, amending 5 U.S.C. §552). 

Available at: < http://frwebgate.access.gpo.gov/cgi- 
bin/getdoc.cgi?dbname=104 cong public laws&docid=f:pub!23 1 . 1 04.pdf 

> 

Application to Federal Information Technology and Web 
Management: 

The Electronic Freedom of Information Act Amendments of 1996 (EFOIA) encourage 
government agencies to use new technology to enhance public access to 
agency records and information. The act amends the FOIA to define 
“record” to mean information in records maintained by an agency in any 
format. Agencies responding to a request for records also must make 
reasonable efforts to search for the records in electronic form or format. 

All agencies are to make available electronically frequently 
requested records obtainable under FOIA and are to maintain an 
Electronic Reading Room. These Electronic Reading Rooms present 
special interest collections as well as “policy statements, administrative 
rulings and manuals, and other materials that affect members of the 
public.” For a report on agency compliance see the OMB Watch study 
report at < http://www.ombwatch.org/site/info/efoia99/efoiareport.html >. 

Excerpt of Original Text from Policy Instrument: 

Section 3. Application of Requirements to Electronic Format 
Information 

(f) For purposes of this section, the term— 

(2) “record” and any other term used in this section in 
reference to information includes any information that would 
be an agency record subject to the requirements of this section 
when maintained by an agency in any format, including an 
electronic format. 

Section 5. Honoring Form or Format Requests 

(C) In responding under this paragraph to a request for 
records, an agency shall make reasonable efforts to search for 
the records in electronic form or format, except when such 
efforts would significantly interfere with the operation of the 
agency's automated information system. 

2.2.4.1.3. Statute : Privacy Act of 1974. (Public Law 93-579, Sec. 3, 
codified as amended at 5 U.S.C. § 552a). 
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Available at: < http://frwebgate.access.gpo.gov/cgi- 
bin/getdoc.cgi?dbname=1994 uscode suppl 4&docid=5usc552a >. 

Application to Federal Information Technology and Web 
Management: 

Congress intended the Privacy Act to protect citizens against 
disclosure of their personal information unless required by the 
Freedom of Information Act, section 552. Congress was especially 
concerned with the automation of information retrieval and 
technological advances in information storage when it enacted this 
legislation. The legislative history of the Privacy Act shows it 
contemplates the problem of individuals’ increased personal exposure 
through the automation of government operations and development of 
computerized government systems. 4 

The Privacy Act describes the manner in which personal 
information of individuals can be disclosed by the government. No 
record containing personal information may be disclosed without the 
prior written consent of the individual to whom the information 
pertains. The act also requires agencies to allow individuals access to 
their records. 

Excerpt of Original Text from Policy Instrument: 

(b) Conditions of Disclosure. 

No agency shall disclose any record which is contained in a system of 

records by any means of communication to any person, or 
to another agency, except pursuant to a written request by, 
or with the prior written consent of, the individual to whom 
the record pertains, unless disclosure of the record would 
be - 



(e) Agency Requirements. - Each agency that maintains a 
system of records shall 

(1) maintain in its records only such information about an 
individual as is relevant and necessary to accomplish a 



4 See Senator Ervin’s speech on June 11, 1974: “It is a rare person who has escaped the quest of modern 
government for information. ... When this quite natural tendency of Government to acquire and keep and 
share information about citizens is enhanced by computer technology and when it is subjected to the 
unrestrained motives of countless political administrators, the resulting threat to individual privacy make it 
necessary for Congress to reaffirm the principle of limited, responsive Government on behalf of freedom.” 
Senate Rpt. (of Government Operations Committee) No. 93-1183, page 6919; Thomas v. U.S. Dept, of 
Energy, 719 F3d 342 (1983).] 
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purpose of the agency required to be accomplished by 
statute or by executive order of the President; 

(2) collect information to the greatest extent practicable 
directly from the subject individual when the information 
may result in adverse determinations about an individual's 
rights, benefits, and privileges under Federal programs; 

(3) inform each individual whom it asks to supply 
information, on the form which it uses to collect the 
information or on a separate form that can be retained by 
the individual. 

2.2.4.1.3.1. Implementing Guidance: Appendix I to OMB 
Circular No. A-130 - Federal Agency Responsibilities for 
Maintaining Records About Individuals 

“This Appendix describes agency responsibilities for 
implementing the reporting and publication requirements of 
the Privacy Act of 1974, 5 U.S.C. 552a, as amended 
(hereinafter "the Act"). It applies to all agencies subject to 
the Act. Note that this Appendix does not rescind other 
guidance OMB has issued to help agencies interpret the 
Privacy Act's provisions, e.g., Privacy Act Guidelines (40 
FR 28949-28978, July 9, 1975), or Final Guidance for 
Conducting Matching Programs (54 FR at 25819, June 19, 

1989).” 

2.2.4.1.3.2. Implementing Guidance: Instructions on complying 
with President’s Memorandum of May 14, 1998, “Privacy and 
Personal Information in Federal Records.” 

Available at: 

<http://www.pub.whitehouse.gOv/urires/I2R7um:pdi://oma.eop.go 
v. us/ 1 998/5/1 4/8 .text. 1 >. 

Application to Federal Information Technology and Web 
Management: 

In this Memorandum the President directed Federal 
agencies to review their current information practices and 
ensure that they are being conducted in accordance with the 
Privacy Act of 1974. The President also directed OMB to issue 
instructions to the agencies on how to conduct this review. 

Excerpt of Original Text from Policy Instrument: 

1 . Designate a Senior Official for Privacy Policy 

Each agency head should have already designated a senior 

official within the agency to assume primary responsibility 



McClure, Sprehe & Eschenfelder 



2-14 



Federal Website Performance Measures 



for privacy policy, in accordance with the President's 
Memorandum. This individual will not necessarily be the 
same person who is responsible for implementation of the 
Privacy Act. For most Cabinet agencies, the appropriate 
official would probably be a policy official at the Assistant 
Secretary level, or equivalent, who in a position within the 
agency to consider privacy policy issues on a national level. 

2. Review and Improve the Management of Privacy Act 
Systems of Records 

Each agency shall conduct a thorough review of its systems 
of records, system of records notices, and routine uses in 
accordance with the criteria and guidance below. Because 
the President directed agencies to review systems of 
records, we have provided guidance on a subset of the 
Privacy Act's requirements that are particularly relevant to 
systems of records. 

2.2.4.1.3.3. Implementing Guidance: M-99-18. Director Jacob 
J. Lew’s Memorandum for the Heads of Executive 
Departments and Agencies on Privacy Policies on Federal Web 
Sites. 1999. 

Available at: < http://cio.gov/docs/webprivl.htm >. 

Application to Federal Information Technology and Web 
Management: 

This memo instructs Departments and Agencies to display 
their privacy policies on their websites, and to comply with various 
laws and regulations governing privacy policies, such as the 
Privacy Act, OMB Circular No. A-130, and Principles for 
Providing and Using Personal Information published by the 
Information Infrastructure Task Force on June 6, 1995. Federal 
agency web developers must include a privacy statement on their 
departmental websites. 

2.2.4.1.3.4. Implementing Guidance: M-00-13, Director Jacob 
J. Lew’s Memorandum for the Heads of Executive 
Departments and Agencies on Privacy Policies and Data 
Collection on Federal Web Sites, June 22, 2000 

Available at: < 

http://www.whitehouse.gov/OMB/memoranda/mOO-13.html> . 

Application to Federal Information Technology and Web 
Management 
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“Particular privacy concerns may be raised when uses of web 
technology can track the activities of users over time and across 
different web sites. . . “Cookies” - small bits of software that are 
placed on a web user’s hard drive - are a principal example of 
current web technology that can be used this way. . . Because of 
the unique laws and traditions about government access to citizens’ 
personal information, the presumption should be that “cookies” 
will not be used a Federal web sites. Under this new Federal 
policy, “cookies” should not be used at Federal web sites, or by 
contractors when operating web sites on behalf of agencies, unless, 
in addition to clear and conspicuous notice, the following 
conditions are met: a compelling need to gather the data on the 
site; appropriate and publicly disclosed privacy safeguards for 
handling of information derived from “cookies”; and personal 
approval by the head of the agency.” 

Also, “"It is federal policy that all Federal web sites and 
contractors when operating on behalf of agencies shall comply 
with the standards set forth in the Children's Online Privacy 
Protection Act of 1998 with respect to the collection of personal 
information online at web sites directed to children." 

2.2.4.I.4. Statute : Public Printing and Documents: Depository 
Library Program (44 U.S.C. 1902). 

Available at: http ://frwebgate 1 . access, gpo . gov/cgi- 

bin/waisgate.cgi?WAISdocID=040147185+0+0+0&WAISaction=r 

etrieve 



Application to Federal Information Technology and Web 
Management: 

Agencies are to make their publications available to depository 
libraries through the Superintendent of Documents. 

Excerpt of Original Text from Policy Instrument: 

Sec. 1902. Government publications, except those 
determined by their issuing components to be required for official 
use only or for strictly administrative or operational purposes 
which have no public interest or educational value and publications 
classified for reasons of national security, shall be made available 
to depository libraries through the facilities of the Superintendent 
of Documents for public information. 

2.2.5. PAPERWORK REDUCTION ACT 



McClure, Sprehe & Eschenfelder 



2-16 



Federal Website Performance Measures 



2.2.5.I. Statute: Paperwork Reduction Act of 1995. (Public Law 104- 
13, codified at 44 U.S.C. 3501 et seq.). 

Available at: 

P.L. 104-13: < http://frwebgate.access.gpo.gov/cgi- 
bin/getdoc.cgi?dbname=104 cong public laws&docid-f:publl3.104. 
pdf >. 

44 U.S.C. 3501: < http://frwebgate.access.gpo.gov/cgi- 
bin/getdoc.cgi?dbname=1994 uscode suppl 3&docid=44usc3501 >. 

Application to Federal Information Technology and Web 
Management: 

The Paperwork Reduction Act (PRA) intends to minimize the 
paperwork burden for individuals, small businesses, educational and 
nonprofit institutions, Federal contractors, state, local, and tribal 
governments, and other persons which results from the collection of 
information by or for the Federal government. PRA is also designed 
to coordinate, integrate, and make uniform Federal information 
resources management policies and practices in order to improve the 
efficiency of government programs. The 1995 amendments to the 
PRA added extensive agency responsibilities for information 
dissemination, responsibilities that are applicable to agency websites. 
Suggested implementation of this law in conjunction with the 
Government Performance and Results Act of 1993 and the Information 
Technology Management Reform Act of 1996 occurs in Executive 
Order 13011, Federal Information Technology. This implementation 
was followed up with another memorandum issued by the President on 
December 1 7, 1 999, Memorandum on E-Government, to improve the 
interoperability of government agencies and the sharing of information 
resources. 

Excerpt of Original Text from Policy Instrument: 

Sec. 2. Coordination of Federal Information Policy. 

Chapter 35 of title 44, United States Code, is amended to 

read as follows: 

* * * 

“Section 3504. Authority and Functions of Director 

“(a)(1) The Director shall oversee the use of 
information resources to improve the efficiency and 
effectiveness of governmental operations to serve agency 
missions, including burden reduction and service delivery 
to the public. In performing such oversight, the Director 
shall — 
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“(A) develop, coordinate and oversee the 
implementation of Federal information resources 
management policies, principles, standards, and guidelines; 
and 

“(B) provide direction and oversee — 

* * * 

“(v) privacy, confidentiality, security, disclosure, 
and sharing of information; and 

“(vi) the acquisition and use of information 

technology. 

* * * 

“Section 3506. Federal Agency Responsibilities 
* * * 

"(d) With respect to information dissemination, each 
agency shall-- 

(1) ensure that the public has timely and equitable 
access to the agency's public information, including 
ensuring such access through- 

(A) encouraging a diversity of public and 
private sources for information based on government public 
information; 

(B) in cases in which the agency provides 
public information maintained in electronic format, 
providing timely and equitable access to the underlying 
data (in whole or in part); and 

(C) agency dissemination of public 
information in an efficient, effective, and economical 
manner; 

(2) regularly solicit and consider public input on the 
agency's information dissemination activities; 

(3) provide adequate notice when initiating, 
substantially modifying, or terminating significant 

information dissemination products;" 

* * * 

Section 3511. Establishment and Operation of the Government 
Information Locator System 
* * * 

(a) In order to assist agencies and the public in 
locating information and to promote information sharing 
and equitable access by the public, the Director shall - (1) 
cause to be established and maintained a distributed 
agency-based electronic Government Information Locator 
Service (hereafter in this section referred to as the 
"Service"), which shall identify the major information 
systems, holdings, and dissemination products of each 
agency; 
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2.2.5.I. Implementing Guidance: Office of Management and 
Budget (OMB), Circular A-130, Management of Federal 
Information Resources. 1996. (61 FR 6428) 

Available at: 

< http://www.whitehouse.gov/OMB/circulars/al 30/al 3Q.html >. 

Application to Federal Information Technology and Web 
Management: 

Circular A-130 provides uniform government- wide 
information resources management policies as required by the 
Paperwork Reduction Act of 1980 and as amended by the 
Paperwork Reduction Act of 1995 (44 U.S.C. 35). This policy 
instrument provides a framework for the development of an 
information management plan to aid Federal agencies in the 
development of their websites. 

Excerpt of Original Text from Policy Instrument: 

8. Policy, a. Information Management Policy 
1. Information Management Planning. Agencies shall 
plan in an integrated manner for managing information 
throughout its life cycle. Agencies shall: 

(a) Consider, at each stage of the information life cycle, 
the effects of decisions and actions on other stages of 
the life cycle, particularly those concerning information 
dissemination 

(e) Integrate planning for information systems with 
plans for resource allocation and use, including 
budgeting, acquisition, and use of information 
technology; 

8. Policy, b. Information Systems and Information 
Technology Management 
1 . Evaluation and Performance Measurement. 

Agencies shall promote the appropriate application of 
Federal information resources as follows: 

(a) Seek opportunities to improve the effectiveness and 
efficiency of government programs through work 
process redesign and the judicious application of 
information technology; 

(b) Prepare, and update as necessary throughout the 
information system life cycle, a benefit-cost analysis for 
each information system: 
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9. Assignment of Responsibilities, c. Department of 
Commerce. The Secretary of Commerce shall: 

4. Conduct studies and evaluations concerning 
telecommunications technology, and concerning the 
improvement, expansion, testing, operation, and use of 
Federal telecommunications systems and advise the 
Director, OMB, and appropriate agencies of the 
recommendations that result from such studies; 

2.2.6. CLINGER-COHEN ACT 

2.2.6.I. Statute: Information Technology Management Reform Act of 
1996. (Public Law 104-106, Division E). 1996. Amended by 
Public Law 104-208, Division A, Title I, Sec. 101(f) (Title VIII, 
Sec. 808(b)), providing that this act (and the Federal 
Acquisition Reform Act of 1996 (Public Law 104-106, Division 
D)) may be cited as the "Clinger-Cohen Act of 1996." 

Available at: 

P.L. 104-106: < http://frwebgate.access.gpo.gov/cgi- 
bin/getdoc.cgi?dbname= : 104 cong public laws&dockfrTpubllOb. 104.pdf 

> 

P.L. 104-208: < http://frwebgate.access.gpo.gov/cgi- 
bin/getdoc.cgi?dbname=104 cong public laws&docid=f:publ208. 104.pdf 

> 

Application to Federal Information Technology and Web 
Management: 

The Clinger-Cohen Act is intended to address the management of IT in the 
Federal government. It is to accomplish this goal through a variety of 
methods, including the use of capital planning for IT acquisitions and 
investments, the establishment of Chief Information Officer (CIO) 
positions in Federal departments and agencies, and the requirement of 
performance measurements of IT. The CIO, through IT, would also be 
responsible for making government more effective, efficient, and 
productive, and include the implementation of policies affecting Federal 
agency Web development. The CIO has agency responsibility for IT 
management and development and reports directly to the head of the 
agency. As regard performance measures, the Clinger-Cohen Act 
represents the specific application of the Government Performance and 
Results Act to the area of IT. 

Excerpt of Original Text from Policy Instrument: 

Section 5112. Capital Planning And Investment Control 

(b) USE OF INFORMATION TECHNOLOGY IN FEDERAL 

PROGRAMS- The Director shall promote and be responsible for 
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improving the acquisition, use, and disposal of information technology by 
the Federal Government to improve the productivity, efficiency, and 
effectiveness of Federal programs, including through dissemination of 
public information and the reduction of information collection burdens on 
the public 

(f) USE OF BEST PRACTICES IN ACQUISITIONS- The Director shall 
encourage the heads of the executive agencies to develop and use the best 
practices in the acquisition of information technology. 

Section 5113. Performance-Based and Results-Based Management 

(a) IN GENERAL- The Director shall encourage the use of performance- 
based and results-based management in fulfilling the responsibilities 
assigned under section 3504(h), of title 44, United States Code. 

(b) Evaluation of Agency Programs and Investments- 

(1) REQUIREMENT- The Director shall evaluate the information 
resources management practices of the executive agencies with respect to 
the performance and results of the investments made by the executive 
agencies in information technology. 

(2) DIRECTION FOR EXECUTIVE AGENCY ACTION- The Director 
shall issue to the head of each executive agency clear and concise 
direction that the head of such agency shall- 

(A) establish effective and efficient capital planning processes for 
selecting, managing, and evaluating the results of all of its major 
investments in information systems 

2.2.6.2. Implementing Guidance: 

2.2.6.2.I. Executive Order 

Federal Information Technology. (Executive Order 13011). 
1996. 

Available at: 

< http://ffwebgate.access.gpo.gov/cgi- 

bin/getdoc.cgi?dbname=l 996 register&docid=fr 19jy96-l 3 3 .pdf > 

Application to Federal Information Technology and Web 
Management: 

This Executive Order links the Clinger-Cohen Act, the PRA, and 
the GPRA. It formalizes OMB’ s oversight of IT management and 
stresses the importance of performance-based planning and 
implementation of Federal IT. Executive Order 1301 1 also creates 
the Chief Information Officer Council, the Government 
Information Technology Services Board, and the Information 
Technology Resources Board. Also falling under the domain of 
this executive order are the establishment of interagency support 
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structures to share IT ideas, minimize duplication, and increase 
interoperability; and design technology procedures and standards. 
The order does not formally define the support structure. Federal 
web site design under this executive order should be interoperable 
and standardized with other agencies. Website design must also 
operate within the realm of the Clinger-Cohen Act, the PRA, and 
the GPRA. The CIO would be in charge of overseeing the design 
with ultimate control still resting with the OMB. 

Excerpt of Original Text from Policy Instrument: 

Section 1 . Policy. It shall be the policy of the United States 
Government that executive agencies shall: * * * 

(b) refocus information technology management to support directly 
their strategic missions, implement an investment review process 
that drives budget formulation and execution for information 
systems, and rethink and restructure the way they perform their 
functions before investing in information technology to support 
that work; 

(c) establish clear accountability for information resources 
management activities by creating agency Chief Information 
Officers(ClOs) with the visibility and management responsibilities 
necessary to advise the agency head on the design, development, 
and implementation of those information systems. These 
responsibilities include: (1) participating in the investment review 
process for information systems; (2) monitoring and evaluating the 
performance of those information systems on the basis of 
applicable performance measures; and, (3) as necessary, advising 
the agency head to modify or terminate those systems. ... 

Section 2. Responsibilities of Agency Heads. The head of each 
executive agency shall: (a) effectively use information technology 
to improve mission performance and service to the public; 

(b) strengthen the quality of decisions about the employment of 
information resources to meet mission needs through integrated 

analysis, planning, budgeting, and evaluation processes, including: 
* * * 

(2) establishing mission-based performance measures for 
information systems investments, aligned with agency 
performance plans prepared pursuant to the Government 
Performance and Results Act of 1993 (Public Law 103-62).... 

2.2.6.2.2. OMB Circular : Proposed Revision of OMB Circular No. A- 
130, Management of Federal Information Resources, April 13, 
2000 
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Available at: 

http://www.whitehouse.gov/omb/federeg/reval30.pdg . 

Application to Federal Information Technology and Web 
Management: 

In proposing to revise OMB Circular No. A- 130, OMB announced 
that its primary reason for doing so was to implement provisions of 
the Clinger-Cohen Act. Agencies must develop annually an 
Information Resources Management Plan. The plan must include 
an annual performance plan as required by GPRA. This must 
include an accountability report, comparing actual performance 
with expected performance. Their benefit-cost analyses of 
information systems must include performance measures. In its 
capital planning process, each agency must institute performance 
measures that monitor actual performance as compared with 
expected results. They must also conduct evaluations of 
information systems and of information resource management 
processes. 



2.2.7. SECURITY 

2.2.7.I. Statute : Computer Security Act of 1987. (Public Law 100-235, 
codified in part as amended at 15 U.S.C. §§ 278g-3, 278g-4). 1988. 

Available at: < http://cio.doe.gov/ucsp/csa.htm >. 

Application to Federal Information Technology and Web 
Management: 

The Computer Security Act of 1987establishes in law for information 
technology the basic principle of security, namely, security commensurate 
with the risk and magnitude of the harm resulting from the loss, misuse, or 
unauthorized access to or modification of information. It also creates a 
means, through the National Institute of Standards and Technology 
(NIST), for establishing minimum acceptable security practices for 
Federal computer systems without limiting the scope of security measures 
already planned or in use. The act requires each agency with a Federal 
computer system to establish a plan for the security and privacy of 
sensitive information, and prescribes “cost effective” rather than 
“absolute” computer security. The NIST is responsible for developing 
standards and guidelines necessary for assuring the protection of sensitive 
information. Federal websites offer one more entry point through which 
security can be breached. Federal CIOs must utilize the most recent 
proven effective hardware and software to protect sensitive departmental 
information and computer systems. 
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Congress is currently considering H.R. 2413, the Computer Security 
Enhancement Act of 1999, available at 
< http://frwebgate.access.gpo.gov/cgi- 

bin/getdoc.cgi?dbname=106 cong bills&docid=f:h2413ih.txt.pdf > This 
bill would enhance the ability of NIST to improve computer security for 
nonclassified information on Federal computer systems. It would also 
promote the use of private sector security technology to protect Federal 
computer systems. Rather than government driving security technology, 
private industry or the market would be responsible for developing 
security technology. 

Excerpt of Original Text from Policy Instrument: 

Section 3. Establishment of Computer Standards Program 

The Act of March 3, 1901, (15 U.S.C. 271-278h), is amended — 

* * * 

(2) by redesignating section 20 as section 22, and by inserting after 
section 19 the following new sections: “SEC. 20. (a) The National 

Bureau of Standards shall — 

* * * 

“(3) have responsibility within the Federal Government for developing 
technical, management, physical, and administrative standards and 
guidelines for the cost-effective security and privacy of sensitive 
information in Federal computer systems except [as specified in the 

law]; 

* * * 

“(6) develop validation procedures for, and evaluate the effectiveness 
of, standards and guidelines developed pursuant to paragraphs (1), (2), 
and (3) of this subsection through research and liaison with other 
government and private agencies.” 

2.2.1. 2. Implementing Guidance 

2.2.7.2.I. OMB Circular No. A-130, Appendix III, Security of 
Federal Automated Information Resources 

Available at: 

< http://www.whitehouse.gov/OMB/circulars/al30/al3Q.html >. 

Application to Federal Information Technology and Web 
Management 

“This Appendix establishes a minimum set of controls to be 
included in Federal automated information security programs; 
assigns Federal agency responsibilities for the security of 
automated information; and links agency automated information 
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security programs and agency management control systems 
established in accordance with OMB Circular No. A- 123. The 
Appendix revises procedures formerly contained in Appendix III to 
OMB Circular No. A-130 (50 FR 52730; December 24, 1985), and 
incorporates requirements of the Computer Security Act of 1987 
(P.L. 100-235) and responsibilities assigned in applicable national 
security directives.” 

2.2. 7.1.2. Presidential Decision Directive 63, “Protecting America’s 
Critical Infrastructures.” 

Available at < http://www.pub.whitehouse.gov/uri- 
res/I2R?um:pdi://oma.eop.gov.us/1998/5/26/l.text.l >. 

Application to Federal Information Technology and Web 
Management: 

This Presidential Directive built on the recommendations of the 
President's Commission on Critical Infrastructure Protection and 
sets a goal of a reliable, interconnected, and secure information 
system infrastructure by the year 2003, and significantly increased 
security to government systems by the year 2000, by: Immediately 
establishing a national center to warn of and respond to attacks. 
Ensuring the capability to protect critical infrastructures from 
intentional acts by 2003. PDD 63 addressed the cyber and physical 
infrastructure vulnerabilities of the Federal government by 
requiring each department and agency to work to reduce its 
exposure to new threats. It set up a new structure to deal with this 
important challenge: a National Coordinator; the National 
Infrastructure Protection Center (NIPC) s; Information Sharing and 
Analysis Centers (ISACs); a National Infrastructure Assurance 
Council; and The Critical Infrastructure Assurance Office to 
provide support to the National Coordinator’s work with 
government agencies and the private sector in developing a 
national plan. The applicability to agency websites is that PPD 63 
is fundamental for website security and effectively requires that 
agencies have performance measures of website security. 

2.2.8. ELECTRONIC COLLECTION AND DIGITAL SIGNATURES 

2.2.8.I. Statute : Government Paperwork Elimination Act. (Public 

Law 105-277, Division C, Title XVII, amending 44 U.S.C.). 1998. 

Available at: 

< http://thomas.loc.gov/cgi- 

bin/bdquery/z?dl05:HR04328:|TOM:/bss/dl05query.html| >. 
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Application to Federal Information Technology and Web 
Management: 

The Government Paperwork Elimination Act (GPEA) provides for Federal 
agencies to give persons who are required to maintain, submit, or disclose 
information the option of doing so electronically, when practicable, by 
October 21, 2003. In addition, GPEA directs that electronic authentication 
(electronic signature) methods be used to verify the identity of the sender 
and the integrity of the electronic content. The Director of the Office of 
Management and Budget is required to develop procedures for the use and 
acceptance of electronic signatures by executive agencies and, in 
cooperation with the National Telecommunications and Information 
Administration, to conduct and report to Congress an ongoing study of the 
use of electronic signatures on paperwork reduction and electronic 
commerce, individual privacy, and the security and authenticity of 
transactions. 

Excerpt of Original Text from Policy Instrument: 

Section 1. Policy 

The Office of Management and Budget is directed to maintain 
compatibility with standards and technology for electronic signatures 
generally used in commerce and industry and by State governments and to 
ensure that electronic signatures are as reliable as is appropriate for the 
purpose in question ant that the electronic record keeping systems readily 
preserve the information submitted. 

Section 2. Procedures 

An agency’s determination of which technology is appropriate for a given 
transaction must include a risk assessment, and an evaluation of targeted 
customer or user needs. Performing a risk assessment to evaluate 
electronic signature alternatives should not be viewed as an isolated 
activity or an end in itself. These agency risk assessments should draw 
from and feed into the interrelated requirements of the Paperwork 
Reduction Act, the Computer Security Act, the Government Performance 
and Results Act, the Clinger-Cohen Act, the Federal Managers Financial 
Integrity Act, and the Chief Financial Officers Act. 

2.2.8.2. Implementing Guidance: Implementation of the Government 
Paperwork Elimination Act. 1999. 

Available at: < http://www.whitehouse.gov/omb/fedreg/gpea2.html >. 

Application to Federal Information Technology and Web 
Management: 

The Government Paperwork Elimination Act provides for Federal 
agencies, by October 21, 2003, to give persons who are required to 
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maintain, submit, or disclose information the option of doing so 
electronically when practicable as a substitute for paper, and to use 
electronic authentication (electronic signature) methods to verify the 
identity of the sender and the integrity of electronic content. The Act 
specifically provides that electronic records and their related electronic 
signatures are not to be denied legal effect, validity, or enforceability 
merely because they are in electronic form. OMB ’s implementation 
of the Act is in two parts. The first part sets forth the policies and 
procedures for implementing the Act, and requesting certain specific 
agencies to provide assistance in particular areas. The second part is 
intended to provide Federal managers with practical implementation 
guidance. 

Excerpt of Original Text from Policy Instrument: 

Section 2. What GPEA procedures should agencies follow? 

a. Accordingly, agencies should develop and implement plans, 
supported by an assessment of whether to use and accept documents in 
electronic form and to engage in electronic transactions. The 
assessment should weigh costs and benefits and involve an appropriate 
risk analysis, recognizing that low-risk information processes may 
need only minimal consideration, while high-risk processes may need 
extensive analysis. 

c. The assessment should develop strategies to mitigate risks and 
maximize benefits in the context of available technologies, and the 
relative total costs and effects of implementing those technologies on 
the program being analyzed. The assessment also should be used to 
develop baselines and verifiable performance measures that track the 
agency's mission, strategic plans, and tactical goals, as required by the 
Clinger-Cohen Act. 

d. In addition to serving as a guide for selecting the most appropriate 
technologies, the assessment of costs and benefits should be designed 
so that it can be used to generate a business case and verifiable return 
on investment to support agency decisions regarding overall 
programmatic direction, investment decisions, and budgetary 
priorities. 

Section 3. How should agencies implement these policies and 
procedures? 

a. To ensure a smooth and cost-effective transition to an electronic 
government that provides improved service to the public, each agency 
must: 

(1) Develop a plan (including a schedule) by October, 2000 that 
provides for continued implementation, by the end of Fiscal Year 
2003, of optional electronic maintenance, submission, or 
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transaction of information when practicable as a substitute for 
paper, including through the use of electronic signatures when 
practicable. A copy of the plan should be provided to OMB. 

(7) Consider the record keeping functionality of any systems that 
store electronic documents and electronic signatures, to ensure 
users have appropriate access to the information and can meet the 
agency's record keeping needs. 



Section 2. Procedures 

b. An agency’s determination of which technology is 
appropriate for a given transaction must include a risk 
assessment, and an evaluation of targeted customer or 
user needs. Performing a risk assessment to evaluate 
electronic signature alternatives should not be viewed as 
an isolated activity or an end in itself. These agency risk 
assessments should draw from and feed into the 
interrelated requirements of the Paperwork Reduction 
Act, the Computer Security Act, the Government 
Performance and Results Act, the Clinger-Cohen Act, 
the Federal Managers Financial Integrity Act, and the 
Chief Financial Officers Act. 

Section 3. Agency Responsibilities 
a. In order to ensure a smooth and cost-effective 
transition to a more electronic government providing 
improved service to the public, each agency shall: 

1 . include in its strategic IT plans supporting program 
responsibilities (required under OMB Circular A-l 1) a 
summary of the agency’s schedule to implement 
optional electronic maintenance, submission, or 
disclosure of information when practicable as a 
substitute for paper, including through the use of 
electronic signatures when practicable, by the end of 
Fiscal Year 2003 (note: agencies need not revise their 
reports on Federal purchasing and payment already 
required by OMB M-99-02, but should include the 
automation of purchasing and payment functions in their 
schedule) 

2.2.9. FEDERAL RECORDS 

2.2.9.I. Statute : Federal Records Act, Title 44 U.S. Code, Chapter 31, 
Records Management by Federal Agencies 
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Available from < http://ffwebgate.access.gpo.gov/cgi- 
bin/getdoc.cgi?dbname=1994 uscode suppl 3&docid=44usc3101 > 



“The head of each Federal agency shall make and preserve records 
containing adequate and proper documentation of the organization, 
functions, policies, decisions, procedures, and essential transactions of the 
agency and designed to furnish the information necessary to protect the 
legal and financial rights of the Government and of persons directly 
affected by the agency's activities. “ 

22.9.2. Implementing Guidance: 

2.2.9.2.1. NARA Bulletin 98-02. 1998 [disposition of electronic 
records]. 

Available at: < http://www.nara.gov/records/policy/b9802.html >. 

Application to Federal Information Technology and Web 
Management: 

NARA Bulletin 98-02 reminds Federal agencies of their 
obligations under Federal law, 36 CFR Part 1234, to provide adequate 
documentation of agency activities, and provides guidance to Federal 
agencies in accomplishing this obligation. Agency CIO's are obligated to 
ensure that website records, web pages, are adequately documented and 
preserved. 

Excerpt of Original Text from Policy Instrument: 

a. Agency heads are required by 44 U.S.C. 3101 to "make and 
preserve records containing adequate and proper 
documentation of the organization, functions, policies, 
decisions, procedures, and essential transactions of the agency. 

. . NARA regulations at 36 CFR Part 1222 specify agency 
record keeping responsibilities, including standards for record 
keeping requirements. NARA regulations at 36 CFR Part 1234 
Subpart C specify standards for managing the creation, use, 
preservation, and disposition of electronic records. 

2.2.9.2.2. NARA Bulletin 99-05. 1999. [Disposition of Electronic 
Records (NARA Bulletin 98-02)] 

Available at: <http://www.nara.gov/records/policy/b9905.html >. 

Application to Federal Information Technology and Web 
Management: 
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NARA Bulletin 99-05 informs Federal agencies that they should 
continue to follow NARA Bulletin 98-02, and reminds agency heads as to 
their responsibilities to maintain adequate documentation of records. 
Although not specifically stated, Web pages should be considered Federal 
agency records, and subject to the same guidelines as print records. 

2.2.9.2.3.General Record Schedule 20 (GRS 20). 1995 version. 
[Electronic Records Workgroup] 

Available at: < http ://ardor.nara. gov/grs/grs20 .html >. 

Application to Federal Information Technology and Web 
Management: 

General Records Schedule 20 (GRS 20) was first published in 
1972 as an attempt to address record management for electronic materials. 
Since then GRS 20 has been rewritten five times with the latest version 
being in 1995. The 1995 version of GRS 20 has been through several 
actions in court (see page ?), and as of March 6, 2000, it still stands. 

The 1995 version of GRS 20 addresses the topic of disposal for 
certain electronic records and the paper or microform records associated 
with them. If a department desires to dispose of electronic records, which 
are not covered by GRS 20, the department must file for authorization 
using SF 1 15s, Request for Records Disposition Authority. Web pages 
and documents associated with Web pages may be Federal records, and if 
so, their disposal should be regulated by NARA. 



2.3. JUDICIAL OPINIONS 

An enormous body of case law exists interpreting certain policy instruments affecting 
aspects of public access to government information, while other policy instruments have 
spawned no case law. The following materials illustrate some courts’ approaches to fine- 
tuning statutory language, with their resulting potential effects on Federal information 
technology management. A survey of all relevant case law is beyond the scope of this 
chapter. 

2.3.1. Freedom of Information Act (FOIA) 

The cases speak to courts’ willingness to apply a liberal interpretation to statutes 
governing public access based on a perceived statutory intent to promote open 
government. Courts check this impulse where the duty to provide this information seems 
onerous for the agency or when the information bears slight resemblance to that 
identified in the statute. 

Cases pertaining to: 
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Freedom of Information Act, Public Law 89-487, codified as amended at 5 U.S.C. 
§ 552. Electronic Freedom of Information Act Amendments of 1996, Public Law 
104-231, amending 5 U.S.C. §552. 

1. Regulation promulgated by National Archives and Records Administration 
(GRS 20) authorizing disposal of electronic government records without 
distinguishing significant records for different treatment violated the Records 
Disposal Act and interfered with plaintiffs right, under EFOIA, to request 
electronic records. Public Citizen v. Carlin, 2 F. Supp 2d 1 (D.D.C. 1997); 
reversed by Public Citizen v. Carlin, 184 F.3d 900 (D.C. Cir. 1997) (agencies 
expected to develop their own website records management policies within 
existing NARA guidelines subject to NARA approval), available at 
< http://www.cadc.uscourts.gov/common/opinions/199908/97-5356a.txt >; 
certiorari denied by Public Citizen v. Carlin, 68 U.S.L.W. 3565, 2000 U.S. 

LEXIS 1744, 2000 WL 240442 (March 6, 2000). 

2. Federal agencies are not required to provide records concerning software over 
which they lack “sufficient control”; software and technical documentation is 
protected by FOIA exemption for trade secrets and commercial or financial 
information. Gilmore v. U.S. Department of Energy, 4 F. Supp. 2d 912, 153 
A.L.R. Fed. 759 (N.D. Cal. 1998). 

3. Federal agencies requested to search “all records” must search relevant 
computer databases to comply with FOIA. Mayock v. Immigration and 
Naturalization Service, 714 F. Supp. 1558 (N.D. Cal. 1989). 

4. Computer backup tapes of electronic messages are records within the meaning 
of FOIA. NARA must change its guidance to Federal agencies on record 
management. Armstrong v. Executive Office of the President, 877 F. Supp. 690 
(D.D.C. 1995); reversed by Armstrong v. Executive Office of the President, 90 
F. 3d 553 (D.C. Cir. 1996); certiorari denied by Armstrong v. Executive Office of 
the President, 520 U.S. 1239 (1997). 

Privacy Act of 1974, Public Law 93-579, Sec. 3, codified as amended at 5 U.S.C. 
§ 552a. 

1. Exemptions to FOIA under the Privacy Act of 1974 pertain to electronic 
records. Manna v. United States Department of Justice, 1994 WL 808070 (D.N.J. 
1994). 

2.3.2. The Copyright Act of 1976 

Case law interpreting the Copyright Act of 1976 is similarly abundant and similarly 
limited as regards Federal information technology and electronic media. Aside from 
disallowing copyright protection for government work, 17 U.S.C. § 105, the statute does 
not directly address the interests and duties of the Federal government in its provisions. 
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The Digital Millennium Copyright Act amended the Copyright Act in 1998 to increase 
protection for authors of work in electronic media. 

Individual Case Laws pertaining to: 

Copyright law is governed by Title 17 of the U.S. code as amended by The 
Digital Millennium Copyright Act, Pub. L. 105-304, 112 Stat. 2860. 

1. Compilations of facts are not protected by copyright under the “sweat-of-the- 
brow” doctrine. Feist Publications v. Rural Telephone Service Co., 499 U.S. 340 
(1991). 

2. Holders of copyright may submit protected work to government agencies 
without fear of losing protection of copyright. Practice Management Information 
Corporation v. American Medical Association, 121 F.3d 516 (9 th Cir. 1997); 

Veeck v. Southern Building Code Congress International, Inc., 49 F. Supp 2d 885, 
(E.D. Tex. 1999). 

2.3.4. Electronic Records 

Since the Iran-Contra scandal in the late 1980s, electronic records have been the subject 
of a series of Federal court cases. The cases arose originally because the White House 
announced its intent to destroy the computer backup tapes containing the electronic mail 
that led to discovery of the Iran-Contra materials. Public interest groups sued to prohibit 
destruction on the grounds that the materials in question were federal records. In 
response to judicial opinions, the National Archives and Records Administration issued 
General Records Schedule 20, Electronic Records (see above). GRS-20 itself then 
became the subject of further litigation. 

A seminal case in the series of litigation was Public Citizen et al. v John Carlin et al., 
U.S. District Court, October 22, 1997. In his opinion, Judge Paul Friedman voided GRS- 
20, stating that the Archivist of the United States had overstepped his authority in 
promulgating GRS-20 and arguing, among other things, that electronic records had 
unique value that was not captured with paper printouts. The U.S. Court of Appeals later 
overturned Friedman’s opinion on August 6, 1999, and the Supreme Court denied 
certiorari on March 6, 2000. 

The opinions cited above are available, together with additional legal resources, at < 
http://www.nara.gov/records/grs20/> . 

2.5. CONCLUSION 

Table 2.1 is a summary presentation of the foregoing elaboration of federal policies 
affecting agency websites. For each topic the table shows the relevant statute, executive 
order, or other key document, plus implementing policy guidance, if any. The right hand 
column summarizes the implications for websites. 



McClure, Sprehe & Eschenfelder 



2-32 



Federal Website Performance Measures 



Table 2.1 Summary of Federal Policies Pertaining to Agency Websites 





Topic 


Statute, Presidential 
Directive, or Other 
Document 


Implementing 

Guidance 


Website Implications 


I. Genera] 

Government 

Policy 


A. Performance 
and Results 


Government Performance 
and Results Act 


OMB Circular A-l I, 
Part II 


Performance plans, goals, and 
measures for agency programs 


B. Customer 
Service 


E.O. 12862, Setting 
Customer Service Standards 




Identify customers, their needs, 
and set standards and 
benchmarks 


C. Accessible 

Information 

Technology 


Rehabilitation Act, section 
508 




Information technology 
accessible to persons with 
disabilities 


D. Electronic 
Government 


Pres. Memo on Electronic 
Government 




Standardized access to and ease 
of finding government 
information, plus privacy and 
security 


II. Federal 

Information 

Policy 


A. National 
Information 
Infrastructure 


Nil Agenda for Action 




Make govt information more 
easily and equitably accessible 


B. Privacy and the 
Nil 


Principles for Providing and 
Using Personal Information 


■ 


Guidelines to personal 
information users and providers 


C. Copyright 


Digital Millennium 
Copyright Act 


■ 


Protecting copyright in 
electronic media 


D. Rights of 
Access to 
Information 


Freedom of Information Act 


■ 


State FOIA procedures on 
websites 


Electronic Freedom of 
Information Act 


■ 


Establish electronic reading 
room on websites 


Privacy Act 


OMB Circular A- 
130, Appendix 1 


Handling of personal 
information 


Pres. Memo on 
Privacy and Personal 
Information in 
Federal Records 


Review privacy policies and 
practices; update notices of 
systems of records 


M-99-1 8 on Privacy 
Policies on Federal 
Websites 


Display privacy policies on 
websites 


M-00-13 on Privacy 
Policies and Data 
Collection on Federal 
Websites 


Discouragement of and 
restrictions on use of “cookies” 
on websites; comply with 
COPPA 


Public Printing and 
Documents: Depository 
Library Program 




Publications provided to 
depository library program. 


E. Paperwork 
Reduction Act 


Paperwork Reduction Act 


OMB Circular A- 130 


Framework for agency 
information management plan, 
including information 
dissemination 


F. Clinger-Cohen 
Act 


Information Technology 
Management Reform Act 


E.O. 13011, Federal 

Information 

Technology 


Websites to be interoperable and 
standardized across government 


G. Security 


Computer Security Act 


Proposed Revision of 
OMB Circular A- 
130, April 13, 2000 


Mission based performance 
measures for information 
systems 


OMB Circular A- 
130, Appendix III 


Security controls for federal 
information systems 






PDD 63, Protecting 
America* Critical 
Infrastructures 


Performance measures for 
website security 


H. Electronic 
Collection and 
Digital Signatures 


Government Paperwork 
Elimination Act 


OMB Notice: 
Implementation of 
the Government 
Paperwork 
Elimination Act 


Increase and encourage 
electronic data collection and 
implement digital signatures 



| . BEST COPY AVAILABLE 
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I. Federal Records 



Federal Records Act 



General Records Provide for management of 

Schedule 20, records created on websites 

Electronic Records, 
and various NARA 
Bulletins 
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CHAPTER 3 



REVIEW OF AGENCY PRACTICES 



This chapter provides an overview of evaluation and performance measures practices at 
the Defense Technical Information Center (DTIC), the Energy Information 
Administration (EIA), and the Government Printing Office (GPO) current as of July 
2000. On March 15 and 16, 2000, the investigators interviewed agency personnel from 
each of the participating federal agencies in site visits to the agencies. Each of these 
agencies supplied documents in answer to the investigators' queries. The agencies also 
reviewed two drafts of the chapter, providing comments the study team incorporated, and 
the investigators met with all three agencies together on July 19, 2000. This chapter is 
the result of the meetings and a compilation and analysis of the documents supplied. 

3.0 BACKGROUND 

The study team completed an initial overview and assessment of the evaluation methods, 
statistics, and performance measures currently in use by GPO Access, DTIC, and EIA. 
From each agency, the study team requested the following: 

VI. Set of documents by which the agency evaluates its website material. 

VII. What types of statistics and performance measures is the agency currently 
using for its website? 

VIII. What overall evaluation methodologies are currently in place 
and how are they conducted/operated? 

IX. Can we determine the types of reports generated by the agency evaluation 
efforts and the success or quality of these reports? 

X. To what degree are information policy instruments/guidelines (as per the 
compilation in I) being considered in the documents and reports? 

To the above questions, the investigators added: 

VIII. Preliminary review of website 

IX. Summary comments 

The objectives of the review and assessment were to: 

• Identify the basic evaluation methodologies, statistics, and performance measures 
currently in use at each of the agencies 

• Provide a composite view of the three agencies that would allow each to compare its 
practices and techniques with others 

• Consider which activities might constitute “best practices” for federal website 
performance measures 

• Establish a first step toward developing statistics and measures for federal website 
performance measures. 
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The review and assessment described in this chapter are based primarily on the materials 
that were provided the study team. Additionally, agency staff from each of the 
participating agencies reviewed and commented on two earlier drafts of this chapter. 



3.1. DEFENSE TECHNICAL INFORMATION CENTER 

DTIC not only publishes its own corporate site but is also responsible for developing and 
implementing approximately 100 websites for external clients throughout the DoD. The 
performance measures discussed in this study report apply only to the DTIC corporate 
site (http://www.dtic.mil) . 

3.1.1. Submitted Document Inventory as of May 6, 2000 

1 . Weekly Server Usage Report 

2. Usage statistics on bandwidth utilization 

3. Email forwarded from Marcie Stone, DTIC, forwarded from Terry Hendrix. Email 
Attachments: 

a) Online service totals 

b) “Metrics and Evaluation Proposal” identifying GPRA instrument and schedule for 
planning evaluation of agency’s effectiveness in disseminating information 

c) online services summary 

d) CENDI Agency Survey Questions for 29 February 2000 

e) Survey for assessing DTIC’s STINET (Science and Technology Information 
Network) service 

f) Chart of queries by month (or year) by type 

g) Bibliography of websites to use for refining “metrics,” website evaluation. 

4. Diskette containing preceding attachments 

3.1.2. What specific types of statistics and performance measures is the 
agency currently using for its website? 

1 . Access Watch - < http://www.dtic.mil/usage/200003 12/DTICALL index.html > 

a) Number of hosts 

b) Percent within DTIC and percent outside DTIC 

c) Variety of file types requested (html, gif, jpg, etc) 

d) Number of errors 

e) Number of megabytes of information passed to users 

f) Average number of html pages/hour 

g) Average number of html pages/day 

h) Total number of html pages requested 

i) Number and percent of html pages requested from within DTIC vs. number and 
percent from outside 

j) Total number of all files requested 

k) Frequency of particular page requests 

l) Number of times website accessed by users from particular domains, hosts, and 
using particular browsers 
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2. Multi Router Traffic Grapher (MRTG) “DTIC 7206 CISCO routers Usage Data” 

a) Fractional DS3 to GTE (15MBit) 

b) T1 to Uunet 

c) GW2 10MBit Ft. Belvoir ITN/NiprNet via DASC 

d) GW1 10MBit Ft. Belvoir ITN/NiprNet via INSCOM 

e) T1 to Pentagon 

f) DTIC 100 MBit LAN on GW 

g) DTIC 100 MBit LAN on GW2 

h) GW1 BBN Router Summation 

i) GW2 UUNet Router Summation 

j) 1st Single Line T1 to BBN - turned off Feb99 

k) 2nd Single Line T1 to BBN - turned off Feb99 

l) Fractional DS3 to BBN 15MBit (serial 4/0) 

■ Daily graph - 5 minute average 

■ Weekly graph - 30 minute average 

■ Monthly graph - 2 hour average 

■ Yearly graph - 1 day average 

m) T1 to UUNet (serial 2/0) 

■ Daily graph - 5 minute average 

■ Weekly graph - 30 minute average 

■ Monthly graph - 2 hour average 

■ Yearly graph - 1 day average 

3. Uncertain source 

a) “onlinetotjanwebdata.txt,” a file comparing usage between DTIC’s online services 
(FTP/Listserv/WWW/fulcrum) and between internal and external users (possibly 
a report generated through Access Watch) 

b) “onlinesumjanwebdata.txt,” a file summarizing who (which hosts?) used DTIC 
services and how many megabytes of information they accessed (that month?) 
(possibly a report generated through Access Watch) 

3.1.3. What overall evaluation methodologies (usability, log analysis, 
customer surveys, etc.) are in place and how are they conducted? 

1. Log analysis (“Access Watch” software, modified for the agency, which records 
usage statistics) 

2. Bandwidth usage (“Multi Router Traffic Grapher”) 

3. Customer survey being conducted currently on Secure STINET 

4. “Keynote” software package, which tracks DTIC’s average response time and 
compares it to 40 well-known websites in metropolitan areas 

3.1.4. Can we determine the types of reports that are generated by the 
evaluation efforts and the “success” or quality of these reports? 

1. Customer survey results are still being compiled. Preliminary results showed 
overall user satisfaction. 

2. Weekly reports are generated by software display usage statistics, and bandwidth 
utilization. 
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3.1.5. To what degree are information policy instruments and guidelines 
being considered in these documents and reports? 

1 . “Metrics and Evaluation Proposal,” refers to requests relating to the Government 
Performance and Results Act (GPRA). 

2. The same proposal refers to “CENDI” agencies, which may wield some influence 
within the participating agencies. 

3. The privacy and security statement on DTIC’s website refers to the NARA 
General Records Schedule 20, alluding to EFOIA and Privacy Act. 

3.1.6. Preliminary Website Evaluation based on McClure 2000 criteria. 

■ Although some DTIC holdings are available to the public, DTIC primarily serves 
federal agency employees, government contractors, potential government 
contractors, and participants in certain federal programs. Information unavailable 
to the public (i.e., available only to those registered for DTIC services) is 
available from NTIS (National Technical Information Service). DTIC provides 
the information necessary to obtain such information to people ineligible to 
register for DTIC services. This limitation is not immediately obvious upon 
reaching the site but is available one layer below under “About DTIC.” DTIC 
could make this plainer so that members of the public who are ineligible for 
registration could find the information more quickly. 

■ The language on the introductory page is very technical, and possibly confusing 
for the general public. Alternatively, DTIC might explain the site contents in 
simpler language so that the general public may more quickly decide whether to 
use the site as a resource. 

■ Fifteen buttons point to important areas of the site, but DTIC might consider 
either an organizing structure or decreasing the number of buttons. 

■ A privacy and security statement appears on the introductory page 

■ General statistics are collected and the agency states it destroys the records it 
collects in accord with the NARA General Records Schedule 20. 

■ The text spans much of the screen and is not separated into smaller units, making 
it difficult to scan for relevant information. 

■ The tone is consistently professional and objective. 

■ Most pages have not been updated within the last three months 

■ All pages include a link to a contact person. 

■ The site lacks an overarching structure to place the contents in logical relation. 
The site includes a lot of information, some of which is redundant. 
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■ The “Find It” function defaults into an alphabetical listing, frequently by 
acronym. A user unfamiliar with the acronyms and other agency nomenclature 
might have a difficult time finding a desired topic and, although the list links to 
another glossary of acronyms, the process is awkward. 

■ Each page contains a heading. As noted before, however, the subject headings are 
confusing. They do not suggest a controlled, logical hierarchy. The relationships 
between the content areas are vague - it is unclear from the structure whether one 
contains part of another or is preliminary to another. 

■ From the materials it appears that statistics on website use are available; however, 
the statistics and website evaluation pages are difficult to find. 

3.1.7. Summary Comments 

The DTIC website acts as a portal for the public to other sites designed for fuller public 
use. Further, it provides fuller service to users who are technologically sophisticated and 
familiar with the agency’s structure, context, and background. 

In the past, DTIC has relied on technical self-evaluation sources such as log file analysis, 
bandwidth usage and tracked web site response time. But recent efforts have also 
included user based evaluation techniques such as the Secure STINET customer 
satisfaction survey. DTIC is also involved in a larger CENDI Agency working group 
effort to incorporate greater use of web evaluation tools. 

Future DTIC evaluation efforts might include some of the following: 

■ Continued and expanded use of user focused evaluation methods such as focus 
group testing, usability testing, and collection and analysis of user comments. 
User based evaluation techniques could be used to develop a more user-centric 
organizing framework for DTIC site. 

■ Design guidelines to control the use of technical language and acronyms without 
accompanying explanations 

B Clearer and easier-to-find information about the restrictions faced by the general 
public on the site, and possible alternative information sources. Currently this 
information is available, but DTIC may consider making it more obvious to the 
user. 
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3.2. ENERGY INFORMATION ADMINISTRATION 

3.2.1. Submitted Document Inventory as of May 6, 2000 

1 . EIA help desk notes, customer comments, raw data 

a) Nov99.htm 

b) Dec99.htm 

c) Jan00.htm 

d) Feb00.htm 

e) Feedback. wpd 

2. Materials related to cognitive analysis of the EIA website 

a) Interview protocol and questions for cognitive analysis 

b) Paper providing background and details of evaluation project, methodology 
used in evaluation. 

c) Specific problems identified during Web evaluation, covers five groups of 
websites. Bulleted list focuses on problems experienced. 

d) Usability study conducted in the Bureau of Labor Statistics Lab. Analysis 
of Results from ELA Web evaluation: Second Pre-test. Includes 
demographics of test subjects, data, results and analysis. 

e) Power point Presentation of Web Cognitive and Usability Evaluation 
Status Report. Outlines goals and objectives of the site. Presents results, 
conclusions and findings. 

3. EIA Webtrends snapshot from March 20, 2000, user statistics, raw 

4. Telephone surveys 

a) 1998 Survey of telephone customers and printed publication subscribers 

b) 2000 Survey of telephone customer satisfaction 

5. Web products profile; number of visits and time of stay in minutes 

6. Sample of Altavista queries to the EIA site, from March 20 2000, raw data 

7. Sample of Google queries to the EIA site, raw data 

8. Information on Channeling 

a) Spreadsheet of web user session data and web file downloads. 

b) Information channels handout: used for strategic planning purposes. 

c) EPSC (Information Products and Services Committee) Report to Senior 
Management - EIA Strategic Plan; provides implementation plan for 
channels concept 

9. Quarterly Report on EIA Performance Measures: deals with citations and data 
integrity and quality issues. Lists results of performance measures that are 
described in document 10, Draft ICSP (Interagency Council on Statistical Policy) 
Statistical Agency Performance (See Performance Measures Section II, IV-9, 
VIII-1) 

10. Draft ICSP Statistical Agency Performance: Draft covers 12 agencies and EIA 
cover sheet provides good reference point. 

3.2.2. What specific types of statistics and performance measures is EIA 
currently using for its website? 
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1. Complete Report: Log Analysis Statistics from WebTrends Software Package 
General statistics : 

a) No. of Hits per Home page 

b) No. of User Sessions 

c) No. of Hits per day 

d) Average session length 

e) Most requested pages-most popular website pages and how often they were 
accessed 

f) Top entry/exit pages- first/last page viewed when entering/leaving site 

g) Most downloaded files-most popular file downloads of website 

h) Activity level by day of week 

i) Most downloaded file types-identifies the downloaded file types and total 
kilobytes downloaded for each file type 

j) Most accessed directories-analyzes accesses to the directories to the site 

k) Top referring sites-identifies the domain names or number of IP addresses 
w/links to site 

l) Top search engines-which search engine referred visitors to site the most 
often 

m) Top search keywords-identifies, for each keyword, which search engines 
led visitors to site 

n) Most used browsers-identifies the most popular WWW browsers used by 
visitors to site 

o) Netscape browsers-breakdown of the various versions of Netscape 
browsers that visitors use 

p) Microsoft Explorer browsers-breakdown of the various versions of 
Explorer browsers visitors use 

q) Visiting spiders-identifies all robots, spiders, crawlers and search services 
visiting site 

r) Most used platforms -identifies operating systems most used by visitors to 
site 

s) Single access pages-pages that visitors access and exit without viewing any 
other page 

t) Most active organizations-companies or organizations that access website 
the most often 

u) Summary of activity by day-outlines general server activity, comparing 
level of activity on weekdays and weekends 

v) Activity level by day of week-shows the activity for each day of the week 
by report period 

w) Activity level by hour of day-shows the most/least active hour of the day 
for the report period 

x) Technical statistics and analysis-total number of hits for the site, how many 
successful/failed, and calculates the percentage of hits that failed 

y) Client errors-type of errors returned by the client accessing the server 

z) Server errors-identifies by type the errors which occurred on the server 

aa) Top referring URLs-provides the full URLs of the sites with links to 

website 
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2. Complete Report: Log Analysis 

Performance Measures from Quarterly Report (See Section IV) 

a) Media Citations 

b) Web use 

c) Briefings to Congress and policymakers 

d) Data accuracy 

e) Surveys of subscribers 

f) Telephone Customer Satisfaction Survey 

3.2.3. What overall evaluation methodologies are in place and how are they 
conducted/operated? 

1 . Web Cognitive and Usability Evaluation 

a) Think Aloud Interviews 

b) Customer feedback 

c) Demographic Questionnaire 

d) Timed exercises 

e) Senior Staff Tested 

2. Customer Comments/Surveys 

a) Telephone customer satisfaction survey 

■ Frequency of contact 

■ Questions asked via telephone 

■ Satisfaction w/customer service and information quality 

■ Product users who used EIA Electronic Products or Services during 
the Past Year 

■ Preferences between paper vs. electronic copy of information 

■ Reasons for Calling National Energy Information Center (NEIC) 

■ Comments from Customers 

o Timeliness 

o Data access on the website 
o Improving website 

b) Feedback observations for February 2000 

3. Emailed Customer Comments (Nov 1999 — Feb 2000) 

a) Website Feedback 

■ November 1999 - Received 4 comments/contents included 

■ December 1999 — Received 1 comment/contents included 
a January 2000 - Received 2 comments/content 

■ February 2000 - Received 2 comments/content 

■ Webmaster mail - comments received by webmaster 

■ November 1999 - Received 21 comments/content 

■ December 1999 -Received 13 comments/content 

■ January 2000 — Received 22 comments/content 

■ February 2000 - Received 24 comments/content 

b) Comments prepared by NEIC staff - Staff prepare summary reports based on 
customer input. The summary reports include some comments verbatim. 

■ November 1999 - Included 10 comments/content 
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■ December 1999 - Included 8 comments/content 

■ January 2000 - Included 10 comments/content 

■ February 2000 - Included 48 comments/content 

3.2.4. Can we determine the types of reports that are generated by the evaluation 
efforts and the “success or quality” of these reports? 

1. Blessing, Colleen; Bradsher-Fredrick, Howard et.al. Cognitive Interviewing: 
Applications to Evaluating the Energy Information Administration’s Web Site. 

2. Finn, Pauline. Analysis of Results from EIA Web Evaluation: Second Pre-Test . 
Usability testing results based on small scale, informal studies. 

3. Blessing, Colleen, et.al. Web Cognitive and Usability Evaluation Status Report. 
Power Point presentation of report No .4 

4. 1998 Survey of Subscribers. 

• 56 percent of respondents prefer paper publication to Web version, 
38 percent prefer Web version 

• Reasons Web version preferred: search capabilities; timely 
information; easier to transfer data to others; don’t need bookshelves 
for storage; cheaper than subscribing. 

• Reasons Print version preferred: don’t like reading large documents 
on the PC; use paper publications when away from the PC; like 
having paper archives; takes too long to download; difficult to 
download; difficult to navigate; Internet access unreliable. 

5. 2000 EIA/NEIC Telephone Customer Satisfaction Survey: Findings concerning 
electronic accessibility of information. 

6. IPSC Report to Senior Management - EIA Strategic Plan 

• Implementation of channel plan. 

8. Performance Measures Quarterly Report 

• Measures address citations, congressional briefings and data quality. 

3.2.5. To what degree are information policy instruments and guidelines (as 
per the compilation) considered in the documents and reports submitted by 
EIA? 

April 27, 2000, Draft Definitions Of Measures, A Supplement To Performance 
Measures. On page 2, Paperwork Reduction Act of 1995 directs agencies to 
achieve reductions in respondent burden of 5 percent per year. 

3.2.6.Preliminary Website Evaluation based on McClure 2000 criteria 

1. Performance measures determining the successful use of the search feature on the 
website. 

• Site uses commercial search engine - Google.com 

• Searches lead to other pages on site; however, a button pointing to 
the EIA home page (i.e. “return to EIA home page”) could be 



McClure, Sprehe & Eschenfelder 



3-9 



Federal Website Performance Measures 



included on those pages to insure the user easy navigation back to 
the home page 

• The “back” button is available but after numerous search strings, it 
may be difficult to use “back” button to get back to home page 

2. Does the website have evaluation information and if so what type and how much? 

• There is no evaluation online available to the general public. 

3. There is contact information for customer feedback: 

a An email address, street address, and phone number is made available 
to people who may have questions about the content of the website. 
infoctr@eia.doe.gov 

b An email address and phone number for the webmaster is made 
available to people who have technical site problems. 
wmaster@eia.doe.gov 

c A feedback button on homepage for sending messages 
d Contact list of personnel at EIA 
e Home page has link to privacy policy 

f Primary and Secondary pages have consistent format and design. 

Pages deeper in the structure do not make it clear that pages are 
official parts of EIA website or if user has left the site, 
g Some of the deeper pages take too long to download 
h Each page lists a modification date. 

i Site contains no pricing information, as EIA does not sell anything, 
j FAQ section 

4. Privacy Statement 

a Contains references to FOIA 

b Explains security practices in place to protect the integrity of the site, 
and detection of individuals trying to hack the site 
c Explains the kinds of information collected and how the information is 
used. 

3.2.7. Summary Overview: 

The task faced by EIA in putting its information on the Web is an organizational 
conundrum. The EIA web committee has the unenviable assignment of posting highly 
diverse technical information from many sources on one website suitable for use by the 
layperson. To this end, and to their credit, the EIA Web committee has recognized the 
need for continual evaluation and redesign of the site in reaction to a constantly changing 
environment. They have conducted studies to aid in the creation of the next version of 
their home page to be unveiled in the near future. The new design takes a channel 
approach, strives for less jargon, and is designed with the layperson in mind. 

To prepare for this new design, EIA chose to perform cognitive usability testing. The 
results from this usability testing suggest that EIA’s current web site design meets key 
usability requirements as most users could find the required information. Tests showed 
no correlations among the following variables: 
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• The users’ level of experience using the Internet and their ability to find the 
correct answer to a test question on the web site 

• The users’ familiarity with energy terms and their ability to find the correct 
answer 

• The degree of frequency of EIA website use and ability to find the correct answer. 

As part of the redesign, EIA may wish to develop basic website design guidelines 
applicable to all EIA sites. EIA will doubtless want its design guidelines to avoid 
stunting office or program creativity or limiting office and program ability to respond to 
changing customer needs. How the guidelines are applied is a matter for agency 
determination. 

EIA already has performance measures in place, but these performance measures have 
not been completely modified to include website services and products. For example, 
EIA has output measures in place for the print environment, but not for the electronic 
environment. Current website performance measures focus on number of downloads 
from website and number of user sessions. 

To expand its toolbox of performance measures the study team suggests the development 
of performance measures based on the following: 

• Identify the existing business processes on the Web, in order to develop 
benchmarks for evaluating these processes. 

• Documents supplied to the study team suggest that EIA could develop 
performance measures for user satisfaction. Possible methodologies include an 
online survey similar to GPO’s, or a survey to electronic newsletter participants. 

• Testing by the study team also suggests the need for performance measures 
related to bandwidth and page load times as some pages were slow to load at both 
peak and non peak usage times. 

• Further analysis of customer comments. EIA summarizes and distributes all 
customer comments. Further analysis might include classifying comments as 
positive or negative with the goal to reduce negative comments, or keeping track 
of the number of comments that led to web site changes with the goal of 
increasing customer suggested changes. 

« Number of customer queries receiving responses within a given time period. The 
support team receives about 30,000 customer contacts each year, including 
telephone, web, mail, and walk-in. The team tries to respond to email contacts 
within 24 hours and answer telephone inquiries immediately. EIA does not 
currently keep data about whether all queries receive responses. 

Existing reference materials listed but not in study team’s possession 

1 . Bradsher-Fredrick, Howard, and Rutchik, Robert. Maximizing Feedback to 
Increase Customer Use of the Energy Information Administration World Wide 
Web Site. (Cited in EIA documents but not provided to study team) 
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2. Bradsher-Fredrick, Howard, and Rutchik, Robert. Customer Use of the Energy 
Information Electronic Products During 1996 (With hints of what to expect in 
1997) . (Cited in EIA documents but not provided to study team) 

Agency did preliminary focus group testing, but provided study team no results from 
actual tests. 



3.3. GOVERNMENT PRINTING OFFICE 

3.3.1. Submitted Document Inventory as of April 8, 2000 

1 . GPO Access Web Committee hierarchy 

2. Video Tape of GPO Access Focus Group 

3. Training Course and Trade Show Evaluations and Evaluation Form 

4. Sampling of Bandwidth Utilization Reports for 9.12 - 9.19, 12.26 - 1.02, 
1.16- 1.23, 1.23 - 1.30 

5. Sample of user comments emailed to web committee 

6. Draft website guidelines for GPO Access 

7. Managing the FDLP Electronic Collection: A Policy Planning Document 

8. Performance Measures: Talking Points 

9. Update Report: The Indexing of GPO Access Web Pages 

10. GPO Access Usage: Executive Summary: List of data showing number 
of searches for databases and the number of retrievals for those searches: 1/2000, 
2/2000 

1 1 . Web Trends: GPO Access usage, prepared by EIDS: 1/2000, 2/2000 

12. Biennial Report to Congress on the Status of GPO Access 12/31/99 

13. GPO Access User Survey Findings of April 18, 2000 

14. GPO Access Open Forum Report, Spring Depository Library Council 
Meeting Newport, RI, Tuesday April 11, 2000 

1 5 . GPO Summary of Search Engines Presentation, Spring Depository 
Library Council Meeting Newport, RI, Tuesday April 12, 2000 

16. GPO Access Open Forum Feedback, Federal Depository Library Spring 
Conference, Wednesday April 14, 1999 

3.3.2.What types of statistics and performance measures is GPO currently using for 
its website? 

1 . Document Retrieval Statistics: In order to measure the units of content 
retrieved by users from GPO Access, GPO decided to use documents 
retrieval. This allows the agency to measure both the units of actual content, 
or documents retrieved, as static html pages and those extracted from 
databases made available on GPO Access. Since this eliminates the non- 
content retrievals for items such as graphics on a web page, GPO believes it is 
far more meaningful than the ubiquitous hit counts employed by many 
websites. 

2. Sampling bandwidth utilization : GPO keeps descriptors of server bandwidth 
utilization. 
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3. Electronic User Comments'. Electronic messages originate from the website 
either through the website’s “Questions or Comments: gpoaccess@gpo.gov ” 
link. Further, GPO uses an Automatic Call Distribution (ACD) System and 
Software, which provides real-time and historical data for statistical 
measuring of performance of GPO Access User Support Team’s telephone 
customer service performance. 

• Electronic messages from comments indicate the page the request 
originated from, the client’s browser; client’s name, client's phone 
number; and the content of the comment. 

• Electronic help or support comments only include name, email address, 
date, and message. 

Note, too, that GPO responds to all email and fax inquiries within 24 hours, 
either with an answer (when possible), or with an acknowledgement of 
message receipt and follow-up answer. GPO also plans to measures to track 
user comments and to analyze their impact on changes made to the site. 

4. Internet search engine retrieval statistics'. GPO Access Usage - Executive 
Summary: GPO staff believe that the ease with which users access the website 
is important, and the ability to find the website using commercial search 
engines is an important factor in accessibility. Therefore, GPO has done 
studies using commercial search engines to see if GPO Access pages are 
covered by commercial search engines, and what terms provide the best 
results for retrieval of GPO Access pages through these search engines. 

5. Biennial Report: 

a) Online Tangible Titles Available through GPO Fiscal Year 1999 

b) Cumulative Online Titles Available through GPO Access: April 1998- 
November 1999 

c) GPO Access user support team requests: March 1997 - October 1999 

d) Expenditures and Operating Costs, Savings and Revenues, p.24. 

6. Statistics generated by the Software Package Webtrends 

a) Server Errors: internal server errors/server. 

b) Page not found Errors: resident server 404 errors 

c) Client Errors: error codes from servers accessing site 

d) Dynamic Pages and Forms Errors 

e) Technical Statistics and Analysis 

• Total hits 

• Successful hits 

• Failed hits 

• Failed hits as percent 

• Cached hits 

• Cached hits as percent 

f) Activity level by hour of the day: bandwidth usage by hour of the day 

g) Activity level by day of the week: bandwidth usage by day of the week 

h) Summary of activity by time increment: bandwidth usage across all servers by 
date. 

i) Summary of activity by report period: allows comparison between weekday 
and weekend activity. 
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j) 

k) 

l ) 

m) 

n) 

o) 

P) 



q) 

r) 

s) 

t) 

u) 

v) 

w) 
X) 



y) 



Hits, and user sessions per organization tag; net, org, mil, com, edu, gov, 
arpnet, international 

Most active organizations: AOL, UUNET, att.net 

Most active countries: identifies countries from which visitors originate and 
which server they enter through. 

Top users: Identifies the domain names from which most requests come from. 
Most popular dynamic pages and forms. 

Most downloaded file types: Most frequently downloaded type of file. Ex gif, 
html, htm, shtml, pdf, jpg, txt, etc. 

Most downloaded files: identifies most popular file downloads, does not count 
if transfer was not completed and does not include files downloaded from a 
database search. See Section II, number 1 for information on GPO document 
download counts. 

Top paths through sites: Identifies entry page and next six pages visited. 

Single access pages: identifies the pages from which a user enters and exits 
from without going to other pages in the site. 

Top exit pages: the pages from which users exited the site. 

Top entry requests: Identifies first page linked to from within the site. 

Top entry pages: First page viewed when a user visits the site. Homepage and 
links that connect with site. 

Least requested pages: Least popular pages on website. If committee thinks 
these pages are important, they need to increase their visibility. 

Most requested pages: most popular websites based on access frequencies. 
Also includes average time a user spends viewing the page. 

Server cluster load balance: compares the performance of the individual web 
servers (five servers) in a cluster based on kilobytes transferred (bandwidth 
load balance) 

General Statistics 

• Number of successful hits for entire site 

• Number of page views 

• Number of user sessions 

• User sessions from U.S. 

• International user sessions 

• User sessions of unknown origin 
« Average number of hits per day 

• Average number of page views per day 

• Average number of user sessions per day 

• Average user session length 

• Number of unique users 

• Number of users who visited once 

• Number of users who visited more than once 



Note that GPO does not use all of the measures produced by WebTrends reports. Instead 
GPO focuses mainly on (1) activity level by hour of the day, (2) most requested pages, 
and (3) server cluster load balance. GPO places greater emphasis on document download 
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statistics. They are more accurate measures of web site usage given the prominence of 
the document databases on GPO Access. 

3.3.3.What overall evaluation methodologies are in place and how are they 
conducted/operated? 

1 . Focus Groups. GPO uses focus groups to gather the opinions of specific 
communities about specific aspects of the GPO site. Following is a brief 
outline of a group session assembled specifically for feedback from the 
Federal Depository Library community, and therefore not necessarily 
representative of all focus groups conducted by the GPO: 

Participants: uncompensated librarians 

Topics covered included, but were not limited to: 

Initial impressions, what specifically do people like? 

Ease of use 
Navigation tools 

2. GPO monitors user feedback through 

• Telephone 4,350 per month (Biennial Report 1999 p.20) 

• Email 2,150 per month (Biennial Report 1999 p.20) 

• Faxes 

• Letters 

3. Online user surveys. GPO has conducted three user surveys, the latest of 
October 1999. Each survey iteration has changed slightly as changes occurred to 
the system. To better keep pace with these changes, GPO intends to conduct 
annual surveys from this point on. The most recent online survey focused on the 
following: 

• Key characteristics of GPO Access users: affiliation with libraries, 
schools, private industry, government etc. 

• How users find GPO Access: internet search, following links 

• Frequency of site usage: yearly, weekly etc. 

• Reasons for using GPO Access: tracking legislation, legal research, 
patron assistance, business decisions, homework, news, academic 
research etc. 

• Finding aids employed: MOCAT, Federal Agency Internet Sites, 
Government Internet Sites by Topic etc. 

• Ease of use of the site 

• Content timeliness 

• Technical specifications of the computers and internet connections 
used to access GPO Access: platform, monitor size, connection speed 

• The survey also asked users to rate GPO user support teams in terms 
of: accessibility, courtesy, product knowledge, technical knowledge, 
complaint resolution and response time 

4. Open public forums: 

• Open attendance (e.g. Search Engines Presentation and GPO 
Access Open Forum at the Spring Depository Library Council 
Meeting) 
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o Forums on several topic areas were held during this 
meeting. 

• Open, unguided discussion of GPO Access 

• Forum announced in the Federal Register and on the GPO Access 
Web site in advance of each session. 

3.3.4. Can we determine the types of reports that are generated by the 

evaluation efforts and the “success or quality” of these reports? 

1 . Document Retrieval reports “ GPO Access Usage report. ” As described in 
Section II, number 1, GPO uses documents retrievals, as its primary measure of 
web site usage. The reports show the number of documents, or units or content, 
that users downloaded from GPO Access. 

2. GPO Access Online User Survey reports. As described in the above section, 
GPO has conducted three on-line user surveys, each resulting in a User Survey 
Findings Report. 

3. Update Report: The Indexing of GPO Access Web Pages 

This GPO report analyzes the ability to retrieve GPO Access Web pages through 
Internet search engines. 

4. Biennial Report to Congress on the Status of GPO Access This is the fourth 
of a series of reports produced every odd year. Most recent -12/31/99 

5. Performance Measures Talking Points. Provides a list of GPO Access 
evaluation goals which GPO is currently working toward achieving 

6. Log Analysis Report uses Webtrends Software package. 

3.3.5. To what degree are information policy instruments and guidelines (as 

per the compilation) considered in the documents and reports 
submitted by GPO? 

1. Draft of website design guidelines for GPO Access, 

a) “Guidelines do not apply directly to the World Wide Web Consortium 
(W3C) Americans with Disabilities Act guidelines 

b) “In accordance with Public Law 103-40, The Government Printing Office 
Electronic Information Enhancement Act of 1993, GPO Access was 
established as a means of enhancing public access to federal electronic 
information. To achieve this objective, GPO has attempted to ensure 
equitable access to public, including those persons with disabilities in 
accordance with standards established by the Americans With Disabilities 
Act (ADA) of 1990.” p.2 

c) Whenever a significant change is made to the site, take and save a “snap- 
shot” of GPO Access: 

d) Mentions archive, permanent information, NARA. Does not mention 
instruments per se, but implies NARA and EFOIA. P.4 

2. Biennial Report 

a) Public Law 103-40, the U.S. Government Printing Office Electronic 
Information Enhancement Act of 1993, which charged the Superintendent 
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of Documents within the Government Printing Office to develop 
mechanisms to enhance public access to a wide range of Federal electronic 
information products, p.l 

b) The Senate Report [103-27] incorporated the Federal Bulletin Board into 
GPO Access p.5. 

c) Title 44 of the U.S. Code (44 USC 191 1) stipulates that public access to 
official government information products disseminated through the FDLP 
must be maintained permanently in regional depository libraries and 
depository libraries not served by a regional depository library. Since 
online products are not physically distributed to depository libraries for 
retention, GPO has assumed responsibility for the provision of permanent 
access to government information products residing on GPO Accessp.8. 

d) Refers to meeting the privacy needs of users, but was instituted before 
requirement by legislative action, p. 15. 

e) The Privacy and Security Notice on GPO Access was posted prior to 
legislative actions that would require all Federal Web sites to provide such 
notices. 

f) Following standards set forth in OMB bulletin 95-01 and the National 
Archives and Records Administration (NARA), in January 1996 GPO 
implemented its Government Information Locator Service (GILS) online. 

3. Managing the FDLP Electronic Collection: A Policy and Planning 

Document 

In several instances this document cites existing laws regarding the scope of 

GPO and the Depository Library program as well as collection development 

and ADA. 

a) Criteria for inclusion of the material into the collection are determined by 
44 U.S.C. §1901-1902 (p. 10, 13)” 

b) “GPO will provide a text-only interface for government information 
products made available through GPO servers whenever technologically 
feasible and cost-effective. This will not only allow GPO to maintain 
compliance with the Americans with Disabilities Act (ADA) but also will 
ensure access to users with limited technological and communications 
capability.” (p.l 3) 

c) “. . .For such electronic products, GPO has assumed the role traditionally 
taken by regional depository libraries as provided in 44 U.S.C. §1911 
(P-9) 

d) “except those determined by their issuing components to be required for 
official use only or for strictly administrative or operational purposes 
which have no public interest or educational value and publications 
classified for reasons of national security.” (44 U.S.C. §1902) (p.9) 

e) “ Based on the mandate of 44 U.S.C. chapters 19 and 41, GPO’s 
responsibility for providing permanent public access is extended to 
electronic government information products that meet the statutory 
definition for government publications.” (p.7) 
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f) “Senate report 103-27 incorporated the Federal Bulletin Board into GPO 
access as well.”(p.6) 

g) “FDLP is codified by 44 U.S.C. Chapter 19. Additional detail concerning 
the Cataloging and Indexing is codified at 44 U.S.C. §1710-1711.” (p.5) 

h) “The Government Printing Office Electronic Information Access 
Enhancement Act of 1993 (Public Law 103-40) charged the 
Superintendent of Documents. ...” (p.5) 

3. Electronic Transition Plan. 

GPO considers Permanent Public Access to be an important performance 
measure for Federal Agency Websites. While GPO Access offers the 
immediacy that the public demands from the Internet, it differs from most 
online services in that it also maintains historical archives of information 
previously available on its servers. Permanent public access to a collection 
of electronic government information products is thus ensured by the 
system's storage facilities, allowing the same historical research that is 
possible through printed media. 

3.3.6. Preliminary Review of website 

• Official pages in the first two layers of the site carry the GPO Access seal. 
Third level pages carry the words “GPO Access” from the seal. Currently the 
only section that does not have the seal is the children’s area. 

• When leaving the official GPO site, the user is notified 

• Few images on site, pages download relatively quickly 

• Text only version available 

• Format consistent from page to page 

• Children’s area clearly marked, downloads a little slower due to more 
graphics 

• Has a “What’s New” section 

• Privacy policy link posted on each page 

• Navigation tools provided, quick links, site map 

• Last update date provided 

• No user statistics posted on the site 

• No access fees 

• Several ways to reach support services 

a) Link at top of page called comments: this provides form to fill in 

b) Help link at top of page, takes user to page with contact information; 
phone, fax, email - gpoaccess@gpo.gov 

c) gpoaccess@gpo. gov also at bottom of each page 

• Provides information about this site link 

3.3.7.Summary Comments 

Initially, GPO established the GPO Access website to fulfill the mandate required by PL 
103-40. Under the provisions of this law it has subsequently evolved to provide public 
access to many key government documents. “As of November 1999, the website 
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provided access to more than 104,000 titles on Government Printing Office servers and 
over 62,000 additional titles through links to other federal agency websites.” (Biennial 
Report 1999 p. 1). GPO considers Permanent Public Access to be an important 
performance measure for federal agency websites. GPO Access maintains historical 
archives of information previously available on its servers, ensuring permanent public 
access to its collection of electronic government information products, and facilitating the 
same historical research that is possible through printed media. 

GPO Access is geared toward meeting the changing needs of the general user 
community. As a content oriented web site, GPO Access primarily measures the number 
of documents retrieved by users as a selection of the units of content being distributed by 
the site. Standard page hit statistics do not capture information about the number of files 
downloaded through a search of the GPO Access databases, which serve as the main 
focus on GPO Access and users’ main source of information. Document retrieval 
statistics more accurately show how many of those documents users accessed and to what 
extent users are making use of the resources on GPO Access. In addition to its use of 
document retrieval statistics, staff constantly evaluate other aspects of the website 
through focus groups, online user surveys, user comments, log analysis, trade shows and 
conferences. Further, GPO conducts user-training sessions. 

GPO has ensured that the information on its website is listed in commercial search 
engines by analyzing search results from commercial search engines, and conducting 
search and retrieval studies using commercial search engines. GPO has contacted the 
commercial search engines to eliminate problems with missing pages, and changed their 
hardware configuration to allow spiders - indexing tools used by commercial search 
engines to add sites to their searchable database - to work better. Also, GPO has added 
keywords and descriptive meta-tags to improve their pages' rankings in retrieval lists for 
engines that take advantage of these tags. 

In addition, GPO has conducted bandwidth usage studies to determine peak user times 
and peak usage pages. They have also changed their hardware to allow equivalent 
distribution of bandwidth among their servers. 

GPO access has a privacy statement posted on its homepage and on its children’s area, 
Ben ’s Guide to Government. The GPO access website maintains design and format 
consistency throughout, except for search pages and search result pages. 

The GPO website has addressed Universal Accessibility in its Website Design 
Guidelines. These guidelines enable GPO Access to comply with section 508 of the 
Rehabilitation Act Amendments of 1998 (36 CFR Part 1194). They include providing 
the information in ASCII text format, using only HTML code, minimizing the use of 
images, provide text alternatives for graphics, and test all pages using a range of 
browsers. 

Future steps to enhance current evaluation activities and web site design might include 
the following: 
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• GPO could enhance evaluation by keeping more exact, categorized figures of user 
comments and customer service responses. GPO could set benchmarks for 
adequate response time, and information usefulness to customers. That is, how 
often are questions answered promptly, adequately and correctly? 

• GPO could enhance the value of training surveys by adding questions pertaining 
to the website to the training questionnaire. 

GPO has a well developed management approach for its website and has committed 
significant effort and thought to evaluating it. 

A recent presentation given by T.C. Evans offered the revised strategy for assessing GPO 
Access (See Appendix B). This strategy could serve as a guideline for other agencies in 
establishing or amending their website development strategies. GPO’s strategy is based 
on nine areas for performance measurements: 

• Legal Mandates: Information Policy Instruments 

• Agency Goals and Objectives 

• Technological Infrastructure 

• Site Content 

• Site Design: Consistency 

• User Support 

• Awareness 

• Usage Metrics 

• Customer Satisfaction. 

The outline referenced in Appendix B is an excellent approach and these nine 
measurement areas should be considered in developing a website evaluation and 
measurement strategy. Although all nine areas may not be of equal importance for 
different agencies in ensuring the citizen access to federal government information, they 
are an excellent starting point. 

3.4. DIVERSITY AND CHALLENGES 

The overview provided in this chapter of the evaluation and measurement efforts at 
DTIC, EIA, and GPO demonstrates fne wide range of strategies for measurement. The 
efforts also suggest that different strategies can be useful in different agency settings. For 
example, some agencies may be better able to employ regular and automatic log analysis 
techniques than to implement full-scale usability studies. Others may find that the 
establishment of formal usability labs to “test” the effectiveness of selected pages and 
content may be more effective. Ultimately, the “best” strategies may be those that best 
accommodate agency mission and available resources. 

The review of agency practices described here may also be useful for other federal 
agencies to compare themselves against. As suggested above, it is not that one agency is 
better or worse than another; rather, that these agencies have taken different approaches 
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to developing evaluation and measurement techniques. A range of issues and situational 
factors (see Chapter 5) affect the success with which any agency can engage in an 
ongoing program of evaluation and measurement. 

Each of these three agencies is in the process of developing and evolving their assessment 
techniques. Within this process each has struggled with a comprehensive planning 
approach to formally establish responsibilities, tasking, schedules, statistics, and 
measures to be employed to assess the quality of agency websites and their usefulness to 
users of that website. 

A common comment heard throughout the federal government is the lack of time, 
resources, and personnel to engage in ongoing assessment of websites. While this 
certainly has been true based on the investigator’s experience, agencies will still need to 
develop “quick and clean” assessment techniques - utilizing not only the techniques and 
measures identified in this chapter but also those proposed in the next. 
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CHAPTER 4 



PROPOSED PERFORMANCE MEASURES 



4.0 INTRODUCTION 

This chapter describes and elaborates on the findings from the assessment of the 
materials collected and reported in Chapter 3 from the three sponsoring agencies 
concerning the types of statistics and performance measures the agencies were using. 
Appendix A, Criteria for the Evaluation of Federal Websites, and Appendix B, Website 
Performance Measurement and Evaluation for GPO Access, supplement the information 
in this chapter about performance measures. 

In interviews and meetings with agency representatives involved in this study, the 
investigators received a wealth of information and suggestions. These materials pertain 
to how the agencies are currently measuring their websites’ performance as well as 
management steps the agencies are taking - or plan to take - to better manage their sites. 
In addition, the policy analysis report in Chapter 2 provides a basis for specific 
recommendations about policy based assessment that federal agency staff should consider 
when assessing their websites. 

The chapter presents a way of organizing and understanding website performance 
measures. As the investigators sifted through the data collected from the agencies and 
from their own analyses, they realized that the term “performance measure” can be 
understood in several ways. Many things lumped under the general heading of 
performance measures might be considered conditions prerequisite to performance 
measures. In other cases, the materials pertained to management or operational 
considerations rather than measurement of the website’s performance. Performance 
measures necessarily imply performance goals and the measures yield indicators of 
progress toward achieving the goals. Websites can be the subject of many measures and 
management activities that are not goal achievement indicators. This point should 
become clearer from the exposition below. 

4.1 LEGAL AND POLICY CONDITIONS AFFECTING AGENCY WEBSITES 

Table 4.1 presents a set of questions or possible measures based on Chapter 2 that the 
investigators consider as prerequisites for performance measures. These are questions 
that all federal webmasters should ask of their websites, irrespective of the specific 
mission served by the agency. That is, they apply to all websites simply because they are 
federal websites and not specifically because they are the site of Agency X or Agency Y. 

In effect, the list in Table 4.1 constitutes a set of performance measures of the “Yes-No” 
variety. The “Yes-No” questions are only a beginning point for an agency in considering 
legal and regulatory conditions. They represent a level of minimal compliance, but do 
not necessarily represent “best practice.” For best practices, agencies need to ask 
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themselves about the quality of their response to the conditions. For example, by now 
every federal agency is likely to have some sort of privacy policy notice on its websites. 
Not every agency, however, has a privacy policy notice that complies with the OMB 
guidance and conformss with the model language OMB provided. Again, most federal 
websites have at least token compliance with the statutory requirement and OMB 
guidance for the Government Information Locator Service. Relatively few have 
thoroughly integrated GILS into the overall design of their websites. Hence, Table 4.1 
should only be thought of as a starting point for performance measures to satisfy legal 
and regulatory conditions. 

Table 4.1. Checklist of Legal and Policy Conditions 





Pertaining to Federal Agency Websites 


Statute/Policy 


Checklist Question 


1. Privacy 


A. Does the website contain a privacy notice that complies with the OMB guidance 
and model language for federal websites? 

B. Does the website avoid the use of “cookies” or observe OMB-stipulated 
restrictions? 

C. Does the website comply with the Children’s Online Privacy Protection Act, 
particularly with regard to collecting personal information from children? (Same as 
9-B below) 


2. Freedom of 
Information Act 


A. Conventional FOIA: Does the website contain clear procedures for requesting 
agency records under FOIA? 

B. Electronic FOIA: Does the website contain an Electronic FOIA Reading Room? 


3. Copyright 


A. Does the website management include provisions to ensure that copyrighted 
materials are not posted without permission from copyright holders? 

B. Reuse Restrictions: Can the site content be freely reused without restriction? 


4. Accessibility 


Does the website make provision for accessibility for persons with disabilities in 
accordance with §508 of Rehabilitation Act? 


5. Security 


Does the website management include adequate provisions for protecting the 
security of the website and other agency information systems? 


6. Paperwork 
Reduction Act 


A. Do information collections undertaken via the website have appropriate OMB 
clearances? 

B. Does the website comply with provisions for the Government Information 
Locator Service? 


7. Depository 
Library Program 


Have publications posted to the website been made available to the Federal 
Depository Library Program? 


8. Government 
Paperwork 


A. Does the website permit and encourage electronic information collection? 

B. Does the website permit use of digital signatures? 



Elimination Act 

9. Federal Records Does the website management include adequate provision for identifying website 
Act records and transferring records to agency record keeping systems? 



10. Access for 
Children 


A. Does the website comply with the President’s April 1997 guidance on 
expanding Internet access for children, parents, and teachers? 

B. Does the website comply with the Children’s Online Privacy Protection Act, 
particularly with regard to collecting personal information from children? (Same as 
1-C above) 
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4.2 MANAGEMENT AND INFRASTRUCTURE FACTORS 



Table 4.2 lists some management and operational factors that affect websites. As with 
legal and regulatory conditions, the list can be considered as performance measures or as 
prerequisites to performance measures. And as with the preceding section, Table 4.2 lists 
factors in a more or less “Yes-No” fashion that represents a minimal level of adequacy. 
The list is once again a starting point for agencies to ask themselves how their websites 
stack up on the factor criteria, and then to go beyond the starting point. Going beyond 
entails asking questions about the quality of agency practices and procedures. 

Table 4.2. Management and Infrastructure Factors 
Pertaining to Federal Agency Websites 



Type of Factor 


Factor 


1. Infrastructure 


A. Does the website have sufficient technology infrastructure (bandwidth, 
server controller array, etc.) to ensure adequate response time? 


B. Does the website have the appropriate software to achieve its purposes? 


C. Does the website have the appropriate level of availability (e.g., 24 hours, 7 
days a week)? 


D. Does the website have adequate steps for emergency preparedness and 
disaster recovery? 


E. Does the website have adequate operating resources (funding and personnel) 
to achieve its purposes? 


F. Are server errors and 404 errors (pages not found errors) at an acceptably 
minimum level? 


2. Management 


A. Does the website comport with industry best practices, such as the World 
Wide Web Federal Consortium’s Home Page Guidelines, as regards content 
design, navigation, organization, style, and markup. 


B. Access control: Does the agency exert management control over who may 
post to the website? 


C. Quality control: Does the agency exert management control over the quality, 
design, and style of website postings? 


D. Awareness and Visibility: Has the agency ensured the site is easily reachable 
by registering with search engines and portals and periodically searching for its 
sites on these engines? 


E. -User Support: Has the agency provided an adequate user support staff for 
technical and content queries? 


F. Security: Is the agency providing adequate security to prevent intrusion into 
the website? 


G. Evaluation: Does the agency regularly evaluate the website through log file 
analysis, focus groups, online user surveys, open public forums, and other 
appropriate methodologies? 


H. Records Management: Has the agency made adequate provision for records 
management on the website and integrated the website into the agency records 
management program? 



As an example, Table 4.2 lists records management and suggests taking and saving of 
website snapshots. It does not ask whether the snapshots are transferred to an approved 
agency recordkeeping system and indeed whether the agency has made any efforts to 
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seek records scheduling approval from the National Archives and Records 
Administration for its website records. 

The quality control questions asked in Table 4.2 are also just a beginning. The questions 
pertain to management control, design, and style of the website. They do not pose 
queries about the quality of content found on the site. The table mentions evaluation 
methodologies but only scratches the surface of this complex and extensive topic. 
Agencies may wish to go beyond the query in Table 4.2 and ask how website evaluation, 
if it exists, dovetails with evaluation of other aspects of evaluating the agency’s 
information dissemination products. 

A website is only one dissemination medium among a range of media the agency uses 
including CD-ROMs, printed products, telephonic information services, fax-on-demand, 
in-person presentations at conferences and workshops, and so forth. A fully integrated 
evaluation system would consider all media as an integrated set so that various products 
and services, offered in several media, supplement and complement one another. The 
foregoing examples serve as hints of the depth and complexity of institutional self- 
questioning that the Government Performance and Results Act envisioned and that 
website performance measures entail. 

4.3 PERFORMANCE MEASURES 

4.3.1. Performance Goals. Broadly speaking, all federal agencies share some 
performance goals, while other goals are specific to individual agencies. The divergence 
of specific goals was apparent even within the three agencies participating in this study. 

GPO is working with interested agencies to facilitate permanent public access 
partnerships for electronic agency information products. Permanent public access means 
that government information products within the scope of the Federal Depository Library 
Program remain available for continuous, no-fee public access. GPO also works with 
partner agencies to provide a full range of development and website hosting services to 
agencies without the necessary resources. 

DTIC, because of its particular mission, aims primarily to serve the defense community, 
meaning the Department of Defense and its contractors; serving the general public is a 
secondary goal, albeit quite important. DTIC also is a center for development, 
implementation and hosting of websites for other DoD organizations, and also provides 
consulting services when requested to do so. 

EIA’s website serves only EIA’s mission as a federal statistical agency and is not 
intended to service other agencies. Whereas dissemination to the general public is a 
secondary goal for DTIC, public access and dissemination of energy information are the 
primary goals of El A, goals aggressively and purposefully pursued. 
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Table 4.3. Performance Measures for Federal Agency Websites 



Performance Goal ] 


Basic Measures 


.»• I Other Agehcy;Specific Measures f * 


Extensiveness: 
Amount or extent 
to which services 
are used 


-Information on content 
unit retrievals; e.g., no. of 
document downloads 
-No. of user sessions per 
time period (not no. of hits) 


-No of User contact sessions 

-Activity levels by time periods 

-Ratio of unique to repeat (2 or more) user sessions 

per time period. 


Efficiency: Use of 
resources in 
providing services 


-Cost of providing website 
session per user 
-Percent of operational 
time when website is not 
available 


-No. of FTE hours or days devoted to website 
creation/ maintenance by size of site in pages 
-Cost per user help session 

-Relation to diminishing costs of other publications 
media (e.g., printing) as indicator that website may 
be replacing other media. 


Effectiveness: How 
well the website 
meets the general 
governmental 
objectives and 
specific agency 
objectives 


-Completeness of coverage 
of agency publications, 
press releases, etc. 

-Degree to which website 
is increasing the timeliness 
of access to agency pubs 


-Permanent public access to agency publications 
-Degree to which GILS is integrated into website 
design/operations 

-Degree to which website shows agency reaching 
new constituent audiences 


Service Quality: 
How well the 
website functions 


-User success rate in 
finding specific 
information in a given time 
period 

-Average time between 
user contact request and 
agency response 
-No. of customer 
complaints/suggestions and 
whether agency action 
results. 


-Whether agency has Help Desk dedicated to its 
website 

-24/7 availability measures 
-Minimal 404 errors 

-Courtesy, helpfulness of user support staff 
-Increase in no. of repeat users per time period 


Usefulness: How 
well the website 
meets the needs of 
users 


-Customer comments plus 
surveys and focus groups 
-Degree to which website 
information increases user 
productivity 

-Degree to which website 
information is incorporated 
into other tasks inside and 
outside the agency 


-Cognitive and Usability Evaluation 
-Measured user satisfaction with: 

♦Clarity of homepage; organization of site 
♦Timeliness of website information 
♦Links to other useful information 
-No. of referrals from other websites and sources of 
referrals 



4.3.2. Performance Measures. Table 4.3 provides selected proposed 
performance measures from the many available in Chapter 3 and the appendices to this 
report. The lists are nonexhaustive and only indicative of a range of measures that can be 
listed. 

Performance goals, like performance measures, shade into one another. For example, the 
goal of effectiveness is how well the website meets the objectives of the agency. One 
objective of the agency is doubtless to serve the needs of its information users in the 
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public, so that effectiveness blurs into the goal of usefulness. Hence, the same measure 
may serve as indicator for two different goals. 

Table 4.3 for the most part leaves out performance measures arising from log server 
statistics and products such as WebTrends. Chapter 3 indicates that the participating 
agencies are already employing these measures to various degrees. The numbers and 
kinds of statistical measures from web log servers are extensive and have been 
adequately treated. Additional detail about log analysis techniques and issues can be 
found in Rubin (2000). 5 

Table 4.3 proposes sets of basic and additional performance measures and should be read 
in conjunction with Tables 4.1 and 4.2. The table suggests a set of basic measures and a 
set of other measures that agencies may choose to adopt for their own purposes. 

In using the measures suggested in Table 4.3, agencies will need to carefullly define the 
measures, operationalize them such that valid and reliable data can be collected, and 
develop standardized procedures for collecting and analyzing the data needed to produce 
the measures. It may be desirable for an interagency working group to develop agreed 
upon definitions and procedures so that the statistics and performance measures can be 
compared across agencies. 

4.4. A BEGINNING APPROACH FOR WEBSITE ASSESSMENT 

This chapter presents a view of website goals and performance from three different 
viewpoints: 

• Legal and regulatory conditions affecting federal websites, summarized in Table 
4.1. 

• Management and infrastructure factors pertaining to federal websites, summarized 
in Table 4.2. 

• Performance measures, in the conventional sense of the term, summarized in 
Table 4.3. 

In conclusion, the investigators offer the following recommendations: 

4. Federal agencies should address the questions in Table 4.1 regarding agency 
websites. These are basic questions about whether the practices of the agencies’ 
sites comply with federal law and policy. 

5. Agencies should consider the questions in Table 4.2 as a useful but 
nonexhaustive checklist of infrastructure and management questions they should 
pose with respect to their websites. 

6. Agencies should ask themselves whether their website performance measures 
programs address the six performance goals listed in Table 4.3 and their 



5 Rubin, Jeffrey. Website Assessment Using Log Analysis: Issues and Techniques, in Charles R. McClure 
and John Carlo Bertot, eds., Evaluating Networked Information Services. Medford, NH: Information 
Today, 2000 (in press). 
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associated basic measures. The investigators do not necessarily recommend that 
websites use the exact measures listed in the table, but they do recommend that 
all agencies employ measures at least similar to these. The Other Agency 
Specific Measures are a list of metrics agencies may choose to adapt to their 
particular goals and purposes for their websites. 

The routine use of measures described in Tables 4.1, 4.2, and 4.3 can assist agencies to 
improve their websites as well as meet requirements for performance assessment as 
stipulated in the Government Performance and Results Act. In using these or other 
measures, agency staff should also consider the issues raised in Chapter 5, following. 
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CHAPTER 5 



ISSUES, RECOMMENDATIONS, AND NEXT STEPS 

5.0 INTRODUCTION 

As suggested in earlier chapters, a number of factors affect the successful management, 
operation, and assessment of federal websites. These factors raise various issues to be 
considered during the evaluation and assessment process of websites. Although site 
visits occurred only at EIA, GPO, and DTIC for this study, the investigators have visited 
and worked with other agencies regarding assessment of their websites. 

Based on this knowledge and experience, this chapter identifies and discusses a number 
of key issues and recommendations that agency officials should consider when assessing 
their websites. The chapter also suggests some possible next steps that can be taken to 
continue the work reported in this study. 

5.1 ISSUES AND RECOMMENDATIONS 

The issues discussed here are not intended to be comprehensive; rather, they suggest 
important concerns that will need to be reviewed, discussed, and resolved at individual 
agencies. The solution to these issues is likely to vary depending on a range of 
situational factors in each agency. 

5.1.1. Top Level Administrative Support for Website Assessment 

While some ongoing assessment of selected and specific aspects of the website can be 
done by individuals who simply have an interest in the website, a serious effort, and one 
that is ongoing, requires top level administrative support. Top level administrative 
support means that senior agency officials commit resources, staff, and moral support to 
those involved in the evaluation assessment. Top level administrative support also 
includes administrative interest in the evaluation effort and the desire to obtain 
information and data that might be used to improve the website. Finally, top 
administrative support means seeing to it that someone takes as a result of the evaluation 
information as a means for continual improvement for that website. 

5.1.2. Ongoing Program of Website Evaluation 

Given the increased dependence of many agencies on disseminating information and 
conducting business via their website, ongoing assessment of the website is essential. An 
ongoing program of website assessment is one that minimally: 

• Has assessment goals and objectives 

• Assigns responsibilities to specific individuals or teams 
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• Operates on a schedule in which specific aspects of the website are regularly 
assessed 

• Uses carefully selected performance measures that yield reliable and valid data 

• Relates website assessment to larger agency performance assessment as 
prescribed in the Government Performance and Results Act 

• Regularly reports assessment findings to agency decision makers and users. 

One-shot assessment of selected components of a website is not a program of assessment. 
Agencies need to establish a regular ongoing program of website assessment that is 
supported by top administrative levels. 

5.1.3. Organizing for Website Assessment . 

Agencies have a number of options for organizing and managing the website assessment 
process. In some instances this responsibility has been lodged in the office of the CIO; in 
others, made a part of the Webmaster activities; in still others, accomplished by a team or 
interoffice committee. There are strengths and weaknesses to each of these approaches 
depending on the agency, but the key point is some clearly defined management structure 
and one person who has responsibility for website assessment. In addition, agency staff 
should know who has specific responsibilities in the assessment process, and other people 
in the organization should know who is responsible for what tasks regarding website 
assessment. 

The experience of the investigators is that oftentimes there are no clear lines of 
responsibility for websites assessment, and those assessments carried out are not made 
known to other agency staff. Clear reporting lines and responsibilities for web 
assessment are essential if successful evaluation is to occur. 

5.1.4. Funding Website Assessment 

The experience of the investigators is that very few agencies provide a regular budget or 
adequate resources to support website assessment and the development of performance 
measures appropriate for ongoing assessment of the website. Often, assessment costs are 
buried in other budget categories or are not formally made available. In other instances, 
evaluation occurs only because one or two staff (typically the webmaster) recognize the 
importance of ongoing assessment and simply “do it” despite the lack of a budget. 

Agencies need to recognize that a formal program of ongoing assessment of the website 
will not occur, or at least will not be very successful, without resources and budgeting 
that supports assessment work. Website assessment requires resources and agencies must 
budget the resources so that those doing the evaluation can plan the approach to be used 
in the assessment. The size of the budget will vary from agency to agency, but a clear 
statement of resources needed and committed to website assessment is essential. 
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5.1.5. Agree on Approach and Measures 

Multiple methods and approaches (log file analysis, usability, surveys, focus groups, etc.) 
can be employed to assess federal websites and multiple measures can be developed to 
describe website activities and quality. Readers should consider Chapter 4 as a starting 
point only. It may be less important which approach and which measures are used than it 
is to have agreement on a program that employs some approach and some measures to 
use. Thus, those responsible for website assessment should agree initially how the 
assessment will be done and which measures will be used. 

One problem observed by the investigators in this area is that there are so many possible 
measures that either the staff are stymied by determining which approach should be used, 
or they collect an endless amount of data in hopes of producing some useful measures 
later. A general rule is to collect only that data that produces measures specifically 
related to mission critical operations and services of the website and collect only that data 
that will be analyzed and reported. There simply is not enough time and resources to 
conduct exhaustively comprehensive website assessments. Thus, consider carefully 
which approaches and which measures will have the greatest payback for (1) determining 
the degree to which mission critical goals and objectives are being met, and (2) 
improving the quality of services to agency users (both within and outside the 
government). 

5.1.6. Training 

To some degree, seat-of-the-pants training can develop some basic skills necessary for 
website development. But increasingly, staff charged with the responsibility of website 
assessment and performance measures development will need training in areas such as: 

• Data collection techniques, e.g., survey development, focus groups, interviews, 
etc. 

• Research methods and measurement scales and techniques 

• Statistical analysis 

• Report generation and use of graphics 

• Use of software and technology such as log file analysis to assist in the 
assessments. 



Other skills and knowledge are likely to be needed in the future as web technology 
evolves. Agencies must realize that an ongoing training process in assessment and 
measurement will be needed. Needs assessment of staff skills and development of a 
training program is an excellent starting point here. Agencies can acquire much of this 
training by having staff participate in activities of organizations such as the World Wide 
Web Federal Consortium and attend the annual FedWeb conferences. 

5.1.7. Recognizing the Policy Context 
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A host of policy guidelines, directives, and requirements affect the manner in which 
federal agencies design, develop, administer, and evaluate their websites. Chapter 2 
provides an overview and assessment of these policies. Many agencies simply are not 
aware of these policies and routinely fail to consider them as they develop and operate 
their websites. 

At one level, failure to recognize these policies and guidelines may cause agencies to 
reinvent the wheel related to website evaluation. Some of these policies and guidelines 
(e.g., privacy protection) offer very useful information on how to design and operate the 
website. At another level, agencies may put themselves at risk by ignoring these policies 
and guidelines. Increasingly oversight efforts from Congress, the General Accounting 
Office, and OMB are being focused on website performance and operations at the 
agencies. 

5.1.8. Sharing Knowledge 

The investigators have observed the efforts of a number of agencies as they experiment 
with web assessment methods, measures, and organization to assess their website. As 
agencies move through this process much is learned, but much stays within the agency 
and is not available for others who are confronting the same issues to learn from and 
incorporate in their assessment efforts. Indeed, the three agencies that participated in this 
study were able to learn much from each other. 

Agencies need to do a better job of sharing their knowledge of website assessment, 
letting other agencies know what works and what does not work, and offering ideas and 
suggestions for new approaches. As part of this study the investigators identified similar 
efforts to develop measures and conduct website assessments at CENDI (an interagency 
science and technology information group that includes Commerce, Energy, National 
Technical Information Service, National Library of Medicine, National Agricultural 
Library, Defense, Interior, and others) and from the ICSP (Interagency Council on 
Statistical Policy). While there are groups such as the Federal Webmasters Forum and 
WWW Federal Consortium that might serve as a mechanism to support such sharing of 
information regarding website assessment, no clearinghouse of information about 
methods, techniques, measures, and strategies for assessing federal websites now exists. 
One of the existing groups might profitably undertake establishing such a clearinghouse. 

5.1.9. A Changing Web Evaluation Context 

The context in which agency officials will collect, analyze, and report measures related to 
websites is considerably different than in the past: 

• Agency staff often do not control access to and use of a range of data that describe 
vendor-supplied information services and resources. Some agencies contract with 
vendors (or other governmental units) that are unwilling or unable to provide the 
types of statistics and use data that officials request. Statistics and measures for 
database use and services, nonetheless, are essential. 
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• The rapidly changing technical nature of the networked environment also affects 
the types of services and resources that can be provided by agency websites. As 
the networked services change, new types of evaluation approaches and measures 
may be needed. 

• The level of effort needed to collect, analyze, and report data to produce 
performance measures for web-based services may be greater than that needed to 
produce more traditional statistics. 

• Software such as web server log analysis provides “automatic” counts and 
statistics of selected services and network activities. One can expect that these 
automatic or software-based approaches will become further refined and 
developed. 

• Networked services, costs, and use may be difficult to “unbundle” if the agency 
obtains these services through a consortium of other governmental units. Costs 
can either be hidden or be extremely difficult to allocate to individual operations 
or agencies. 

Agency officials may be entering a period of time where measures for website operations, 
use, and services may be useful for two to four years, perhaps less, and then will have to 
be redeveloped or discarded. Such an environment is quite different than the statistics 
collecting environment in which many agencies previously existed. Despite these 
concerns and factors, agencies can move forward and learn how best to produce and use 
such measures in this new environment. 



5.2. NEXT STEPS 

Federal agency website evaluation and development of performance measures is in its 
infancy and much work lies ahead in designing, testing, and implementing evaluation 
methods and measures. The investigators recommend two broad areas for next steps that 
can build upon the work described in this report. 

5.2.1. Follow-on Study 

The first area is to continue study on a number of topics that have been identified in this 
report. These topics include, but are not limited to: 

• Design and test additional performance measures for agency websites 

• Determine the degree to which website performance measures (as suggested in 
Chapter 4) can be used in agencies beyond EIA, GPO, and DTIC 

• Assess organizational structure and managerial strategies as they affect website 
assessment 

• Identify, describe, and assess costs associated with different levels of web-based 
services 

• Integrate and coordinate various data collection methods and approaches into 
models of evaluation that could be used by agencies at different stages in website 
evaluation 



McClure and Sprehe 



5-5 



Federal Website Performance Measures 



• Assess the way in which agencies have complied with policies affecting website 
development and operation 

Appendix C provides an example proposal for studying the degree to which agencies are 
complying with website policies. The investigators can also develop proposals that 
address other topics identified above. 

An important follow-on to the study reported here would be to determine agency 
compliance with the various policies affecting websites as summarized in Chapter 2. 

Such a study would have a number of benefits for an agency, not the least of which 
would be overall improvement and coordination of its website procedures with the 
ultimate goal of better meeting user needs. 

5.2.2. Disseminate Findings Widely and Increase Awareness 

A second key area for next steps is to disseminate study findings widely and increase the 
awareness and knowledge base of agencies related to website assessment. The findings 
presented in this report might be best distributed by holding a government-wide 
conference for federal webmasters, IT systems experts, policymakers, records managers, 
and others. At this conference agency officials could learn about possible performance 
measures, assessment techniques, and new developments related to agency best practice 
in this area. They could also share their knowledge of assessment as done in their 
agencies. Such a conference would also provide an excellent opportunity for agency 
officials to interact and exchange views on assessment. 

The report presented here might also be formally published by the GPO or another 
federal agency and distributed widely. As suggested earlier in this chapter, there is a 
dearth of information about federal website evaluation and performance measure 
assessment. Wide dissemination of a published report should increase attention to the 
topic and provide others with information that may assist them in their efforts to evaluate 
websites. 

Finally, there needs to be better organization and sharing of knowledge related to website 
assessment and performance measures. An important next step would be to formally 
charter an agency or interagency group (e.g., Federal Webmasters Forum) to serve as a 
clearinghouse on information, reports, best practices, etc. related to assessing federal 
websites. Such an effort could have significant impact for agencies by saving them time 
and effort to develop and refine methods and measures that may already be known and 
tested. 



5.2.3. Continual Improvement of Federal Websites 

In the near term, it is clear that federal agencies will continue to develop and expand their 
web presence. Not only will a significant increase in resources and information be added 
to these websites, but also interactive services and e-commerce will increasingly be 
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conducted with both citizens and with other agencies. Given the government’s ongoing 
efforts to reduce costs and improve services, one can also expect increased policy 
emphasis that encourages agencies to conduct business electronically and via their 
websites. 

Evaluation and performance measures to describe federal websites and resources in the 
networked environment are important tools. But an ongoing program of evaluation that 
regularly produces these performance measures is a tool that is essential for the long-term 
success of the agency. These measures provide agency decision makers with a means to: 

• Describe the degree to which web-based services and resources are accomplishing 
agency goals and objectives; 

• Assess the degree to which resource allocations for various web-based services 
and resources are appropriate; 

• Monitor the status and development of these services so that it is possible to make 
quick corrective actions; 

• Identify the impact, benefits, importance, and problems with web-based services 
and resources; 

• Assist decision-makers to determine whether to continue or modify existing web- 
based services or develop new services; 

• Provide objective information for justification of the website or otherwise 
demonstrate accountability for the web-based programs; and 

• Educate decision-makers, staff, policymakers, users, and others as to the 
importance and benefits from web-based activities and services. 

Perhaps most importantly, an ongoing program of evaluation contributes to the process of 
constant improvement - looking for ways to improve the usefulness, impact, and benefits 
that can result from web-based resources and services. 

At issue is the degree to which these web-based resources and services are cost effective, 
deliver high-quality services, meet the needs of users, comply with existing policy, 
reduce agency costs, and help accomplish agency mission and objectives - to name a few 
criteria. If agencies are to operate successfully in this web-based information 
environment they must assess their success in addressing these criteria in a regular and 
ongoing program of evaluation and measurement. Developing and implementing such a 
program is both doable and essential if the long-term health of the website and the 
agency’s programs are to be insured. 
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APPENDIX A 



CRITERIA FOR THE EVALUATION OF FEDERAL WEBSITES 



NOTE: The materials in Appendix A were developed by 
Charles R. McClure for a class recently taught at Florida 
State University. This appendix represents a slightly 
different approach to many of the subjects addressed in 
this report. Nonetheless, the criteria listed here can be a 
useful tool to assess Federal websites. Hence, the authors 
have included the appendix for its potential usefulness to 
readers of the report. 
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APPENDIX A 



CRITERIA FOR THE EVALUATION OF FEDERAL WEBSITES 



Introduction: Listed below are a number of possible criteria that can be used to assess websites, 
and to some degree, other types of electronic services provision in a networked environment. All 
of these criteria may not be appropriate for specific types of websites. The purpose of the 
website (e.g., educational, entertainment, commercial, services provision, etc.) shapes the types 
of crtieria that may be appropriate for a particular website. 



I. INFORMATION CONTENT CRITERIA 

This section evaluates the substantive aspects of the website 

Orientation to website 

• A website overview is provided (is it clear if this is a commercial, educational, advocacy, 
informational, or other type of website?) 

• The scope of website is clearly stated. 

• The services and information provided at the website are described. 

• Instructions on the use of the website are provided. 

• The purpose/mission should be appropriate to the targeted audience(s). 

• A liability statement warning the user of information provided through the links is provided 
(e.g., access by children). 

• Copyright statements are provided if necessary. 

Content 

• The content of the homepage should match the purpose/mission. 

• The content and links match the needs of the expected audience. 

• The website includes only necessary and useful information, content coverage does not 
overlap. 

• The amount of information is significant and not overwhelming, is balanced throughout the 
website. 

• There are full text references or other resources available. 

• The content has rich and unique quality that inspires users to visit regularly for 

information. 

• The content is written in a clear and consistent language style that matches the 

expected audience. 

• Avoids jargon, humor, condescension, accusation, and chit chat. 

• Uses a positive and professional tone. 
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• Language does not show bias. 

• Content is organized logically throughout the website and by the user's needs. 

Currency 

• Address of contact person and last update information appears at the bottom of pages with 
substantive content. 

• Pages should be kept up to date. 

• The content provided by the website creators is up-to-date. 



Bibliographic Control 

• Headings are clearly phrased, descriptive and understandable. 

• Each screen is titled clearly. 

• If the headings cannot be completely descriptive, coherent and concise, descriptions follow. 

• The information that is provided through the link matches the headings and descriptions. 

• Terminology and layout are consistent within the headings throughout the website. 

• Navigational tools standardized and on each page of site. 

Services 

• Provision of services is different than provision of information resources, (e.g., leaving a 
question to be answered by the reference staff is a service). 

• Are the services open to everyone on the Internet or do parts require fees? 

• Services meet the needs of the user. 

• Services are fully operational and if they cost money are the fees clearly presented? 

Accuracy 

• Statement of document/website status is provided (e.g., if in progress will note, "under 
construction"). 

« Reference or sources of information cited are accurate. 

• Typing, spelling, and grammar errors and other inconsistencies are absent. 

Privacy 

• Degree to which the site provides users with explicit policy on how users’ privacy rights 
are protected. 

• Degree to which the site provides users with information about making public site-use 
information, repackaging, or selling such information to others. 

• Can the user exchange encrypted information with the site? 
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Security 



• Is the user assured that his/her use of the website is scure? 

• Is encrypted use of the website possible? 

• Are passwords needed to access certain parts of the website? 

Availability of Log File Information 

• Do users have access to basic statistics describing use of the website? 

• What information does the website make publicly available regarding the use and assessment 
of the website? 

Retrieval/Search Engine 

• Does the website have its own search engine? 

• Are their instructions to help the user better understand how to do searching on the website? 

• What is the overall effectiveness, ease of use, and power of the search engine? 



Policy Issues 

• Written policies and procedures that govern the development, use, and access to the website 
(who is responsible for what, marketing, commercial use of the site, access by children, etc.? 

• Use of Electronic Records Management policies that guide the management of the website 
(length of time material is on the website, approval for placing material on the website, legal 
liability, acceptable as evidence ina court of law. etc.? 

• Preservation, archival, and scheduling of web content for future access and preservation? 

• Privacy, FOIA, and other federal guidelines and laws? 

• Degree to which organizational policies support broader institutional policies? 



II. EASE OF USE CRITERIA 

This section evaluates the physical movement through the website. 

Quality of Links 

• There are no dead-end links. 

• Temporary forwarding addresses do not qualify as good links. 

• "What's new" section provided for new links - (good for frequent users). 

• Shortcut links are possible for frequent users. 
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• Warning statements are provided, if link will lead to large document or image. 

• Indication of restricted access for a link is provided. 

• Links are provided to mention documents. 

Speed 

• Minimal use of large graphics and bright colors so that document loading is relatively fast at 
slow connection speeds? 

• Efficient programming of links for quick movement through the website? 

Feedback Mechanisms 

• On the homepage and other significant pages there is a contact person: name, address and 
email address of the responsible person or entity for example, "webadmin@library.org." 

• Feedback links are fully operational. 

Accessibility 

• There is an awareness of the existence of this website through search engines or other 
publicity. 

• Full name of website organization is provided in the title, heading, document address, 
graphical link and/or URL in order that the source can be recoverable. 

• It is usually possible to reach the site. It is not frequently overloaded. 

• The URL is not likely to be confused or mistyped. 

• Consistent metadata embedded in pages to improve search engine indexing. 

Design 

• Format is appropriate to subject matter and functionality. A good design directs users toward 
information rather than away from it. 

• The screens are uncluttered. 

• The format is consistent throughout the website. 

• The homepage for the website is short and simple. 

• The website is written in standard HTML language. The site is consistent when 

accessed via different browsers (text and graphic). 

• Graphics and color are used to lead the user through the information appropriately. 

• Monster graphics are not used. 

• Many little graphics are avoided. 

• Use of bold, italics, blinking and other attention getting devices is appropriate. 

• Icons are understandable and make sense as what they actually do. 

• User has the option of turning off the automatic loading of graphics. 
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Navigability 



• Essential instructions appear before links requiring user interaction (e.g., email). 

• Navigation options are distinct and spelled out. 

• Minimal user skills are required. 

• All the parts work. 

• Interactions are secured if they involve private information. 

• Links are provided to return to website homepage on all supporting pages. 

• Links are provided to assist navigation (e.g., "return to top," "return to previous page"). 

Video and Audio 

• Applications on the site are appropriate. 

• Applications work as intended and instructions to operate are clear and straight-forward 

• Audio and video uses provide links to sites where the necessary “plug and play” software 
be obtained and downloaded for immediate use. 

• Enhances site performance and services and is not “glitz.” 
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APPENDIX B 



WEB SITE PERFORMANCE MEASUREMENT 
AND EVALUATION FOR 
GPO ACCESS 



A PowerPoint presentation by 
T.C. Evans 

Given at FedWeb 2000 
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Web Site Performance 
Measurement and Evaluation 
for GPO Access 



Presented to FEDWEB 2000 
By T.C Evans 
May 8, 2000 
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• To enumerate the key areas of performance 
measurement, as \vc see tliem 

• To give sped tic examples of how we evaluate 
these factors for GPO Access 



Slide 3 




BEST COPY AVAILABLE 



McClure, Sprehe & Eschenfelder B-2 Federal Website Performance Measures 

ERIC 



105 




Slide 5 




l • ga iMandJJi |/a iG'fjOMc cessM 

Public Law 103-40 



Online Federal Register and Congressional 
Record ; other databases determined by 
Superintendent of Documents 
A Federal information locator 
Electronic Storage Facility for Federal 
Information 

Federal Bulletin Board (included by 
Senate Report) 
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Improve access to Federal Government 
information in electronic formats 
Provide well-designed, easy to use, publicly 
available online services 
Provide official and complete information 
Provide Finding Aids 

Ensure access to broadest audience possible 
Electronic information a byproduct of 
print process 

Ensure Permanent Public Access 
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Server Controller Array 
Server farm 
Distributed storage 
Permanent data backup 

•-Bandwidth. 

• Partnerships with Dcpts. of Energy, State. 
■and-GommercerkSUrothcrs— 
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Almost 1 .500 databases; over 80 applications 

Over 1 76.000 titles 

Suite of Finding Aids 

Hosted sites lor outside agencies 

Filtered into partnerships 

Secure Online Bookstore 

lien’s Guide to U.S. Government lor Kids 

Commitment to Permanent Public Access 
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Level 2 of W3C Web Design Guidelines 

Simplified URLs 

Consistent look and feel 

Consistent navigational tools 

Site designed to accommodate "lowest 

common denominator'* technology 

dial-up/teluet access 
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• Help available via phone, fax. and e-mail 

• Live phone help available on weekdays 

• Timely response 

• User contacts monitored for quality 
Variety!) f online sel Hiclp materials 
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Awareness 



Reaching intended audience 
Educating audience 
Expanding audience 
Facilitating visibility through 
search engines/portals 
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• Promotional efforts 

• Training classes 

• Trade shows and conference exhibits 

• Speeches and demonstrations 

• Ongoing search engine ranking 
improvement project 
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• Measure content retrieval 

• Measure retrieval for all GPO Access 



components 

• Identity and understand gaps in usage data 

• Content retrieval statistics reported on a 
monthly basis 

• Provide retrieval statistics to hosted site 
ageneies 
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Understanding of user expectations 

- ease of use 

- timeliness 

- responsiveness 

- intuitive site organization 
Meeting of users' expectations 
Problem resolution 
Courtesy 
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EXAMPLE: PROPOSAL TO STUDY AGENCY COMPLIANCE 
WITH FEDERAL WEBSITE POLICIES 
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Example: Proposal to Study Agency Compliance 
with Federal Website Policies 

Study Objectives 

The overall purpose of the proposed project is to better understand the degree to which Federal 
agencies are aware of and have responded to current information policies that affect the 
management and operation of Federal websites. More specifically, study objectives include the 
following: 

• Determine the degree to which a selection of agency web managers and related staff are 
aware of the various information policies affecting the operation and management of 
these websites. 

• Determine the degree to which a selection of agency websites have responded to and 
complied with these various information policies in the management and operation of 
their websites. 

• Assess the degree to which the various information policies actually assist agencies in 
operating successful websites and propose possible policies that should be considered for 
improved management and operation of their websites. 

• Make specific recommendations to the agency as to how its websites might better 
respond to and address the various information policies currently in place. 

Ultimately, the study will assist agencies to determine how well they are responding to the 
existing information policies that affect the operation and management of Federal websites and 
improve the usefulness, impact, and management of those websites. 



Timeline and Tasking 

The study would be a five to seven month effort depending on the depth of investigation require 
by an agency or group of agencies. It would have the following general phases and tasking: 

Phase I: Project Organization (one month). In this first phase the investigators will 
develop detailed project tasking. They will meet with appropriate agency staff to obtain a 
better understanding and knowledge of their websites. Agency staff will provide the 
investigators with a range of internal documents and reports related to their website 
management. The investigators will (with the assistance of agency staff) identify a sample of 
agency websites to be used for the assessment. And the agencies each will establish a single 
point of contact for the study as well as a small advisory committee that can assist in study 
activities and logistics. 
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Phase II: Data Collection (two-four months). During this phase the investigators will use a 
range of methods to obtain data that will answer the study objectives outlined above. These 
methods may include interviews with key informants at participating agencies; log analysis; 
policy analysis; surveys; focus groups; usability assessments; and content analysis of 
websites. The investigators will provide a draft report of preliminary findings at the 
conclusion of this phase. 

Phase III: Final Report, Recommendations, and Dissemination (two months). Based on 
the findings from the data collection, the investigators will produce a final report that will 
summarize the degree to which the websites successfully respond to the information policies 
as outlined in Chapter 2. The report will also offer specific recommendations for (1) how 
these policies might be improved, and (2) how the agency can better utilize these policies for 
improved management and operation of their websites. The investigators will also, with the 
assistance of participating agencies, disseminate project findings and recommendations 
within the agency and the larger Federal community via an Executive Briefing session. 

These proposed Phases and general tasking are likely to be modified and refined to best meet the 
needs of the participating agencies during Phase I of the study. 

Budget 

The investigators propose a project budget in the range of $60,000 - $75,000 for this five-seven 
month study. McClure and Sprehe, as Co-Principal Investigators would contribute a 
predetermined number of days per month on the project. They would hire additional staff, 2-3 
graduate research assistants at 20 hours per week. Other resources would be used for travel, 
communications, supplies, copying, etc. This budget assumes that agency liaisons will be able to 
work with the investigators and can draw upon some assistance from agency staff in the 
completion of the study. 



Study Products 

A number of study products will result from this effort, including: 

• A final report that offers specific recommendations to the participating agencies on how 
they might better respond to and comply with various information policies for improved 
management of and services from their websites. 

• The report will also provide a set of procedures that the agencies can use in the future to 
develop an ongoing program to insure adequate awareness of and compliance with new 
information policies affecting website management and operation. 

• A revised and expanded Chapter 2 ( Information Policies Affecting Federal Websites) 
based on the findings from the study. 
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• An Executive Briefing session for selected agency officials that would increase agency 
knowledge of policies affecting the management and operation of Federal websites and 
how those websites might be improved. 

These products provide an important set of tools and strategies by which agencies can continue 
to improve and develop their websites. 



Importance of the Study 

The investigators’ work in Chapter 2, Information Policies Affecting Federal Websites will 
provide an important tool to agencies that can assist them to better manage and operate their 
websites - and be in compliance with the existing policies affecting website management. The 
procedures and findings from the study are also likely to benefit other agencies that are learning 
how to deal with the various information policies affecting website development. 

In addition, the study will significantly raise the visibility of the importance and use of agency 
websites. The project will also demonstrate how agencies are taking a leadership position in 
addressing the range of Federal information policies that affect website management and 
operations. Finally, and perhaps most importantly, the study will result in agency websites that 
are more successful in meeting agency mission and goals, more successful in meeting user 
information needs, and more successful in how they are managed and operated. 
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